Sep 05 07:48:17.259621 ip-10-0-143-184 systemd[1]: Starting Kubernetes Kubelet... Sep 05 07:48:17.710410 ip-10-0-143-184 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 05 07:48:17.710410 ip-10-0-143-184 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 05 07:48:17.710410 ip-10-0-143-184 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 05 07:48:17.710410 ip-10-0-143-184 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 05 07:48:17.710410 ip-10-0-143-184 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 05 07:48:17.711477 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.711392 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 05 07:48:17.716950 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716923 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 07:48:17.716950 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716943 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 07:48:17.716950 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716947 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 07:48:17.716950 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716950 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 07:48:17.716950 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716953 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 07:48:17.716950 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716956 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 07:48:17.716950 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716959 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716962 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716965 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716968 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716971 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716974 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716976 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716979 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716981 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716986 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716992 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716996 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.716999 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717003 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717006 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717008 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717011 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717014 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717017 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 07:48:17.717223 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717019 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717022 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717024 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717027 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717029 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717032 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717042 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717045 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717047 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717050 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717053 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717055 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717058 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717060 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717062 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717066 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717068 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717071 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717075 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717078 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 07:48:17.717698 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717080 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717083 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717085 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717088 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717090 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717095 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717097 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717100 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717103 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717105 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717108 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717110 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717113 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717115 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717118 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717123 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717126 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717129 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717132 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717134 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 07:48:17.718232 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717136 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717140 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717142 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717145 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717147 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717149 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717151 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717160 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717162 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717165 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717168 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717171 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717174 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717177 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717179 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717182 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717185 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717187 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717190 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717193 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 07:48:17.718710 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717196 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717198 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717201 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717203 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717206 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717225 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717228 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717231 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717233 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717236 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717239 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717241 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717244 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.717246 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718007 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718015 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718018 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718021 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718025 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718029 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 07:48:17.719189 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718032 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718034 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718037 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718040 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718042 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718045 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718047 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718050 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718052 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718055 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718057 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718061 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718064 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718066 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718069 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718071 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718074 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718077 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718079 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718082 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 07:48:17.719722 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718084 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718087 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718089 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718092 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718094 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718097 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718099 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718102 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718104 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718109 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718112 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718116 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718119 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718123 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718125 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718128 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718131 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718133 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718136 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 07:48:17.720386 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718138 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718141 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718143 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718146 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718148 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718151 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718154 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718157 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718159 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718162 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718165 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718167 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718170 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718173 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718176 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718178 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718181 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718183 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718186 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718188 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 07:48:17.720857 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718191 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718193 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718196 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718199 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718201 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718203 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718226 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718231 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718235 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718239 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718242 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718244 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718247 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718249 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718252 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718254 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718257 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718259 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718263 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 07:48:17.721361 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718266 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718272 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718275 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718279 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718281 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718284 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718288 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718291 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718294 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718296 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718299 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718301 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718304 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718306 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718309 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718694 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718698 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718701 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718704 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 07:48:17.721835 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718707 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718709 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718712 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718715 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718717 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718720 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718723 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718725 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718728 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718730 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718733 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718736 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718738 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718741 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718745 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718748 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718751 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718753 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718756 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718758 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 07:48:17.722317 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718761 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718763 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718765 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718768 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718770 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718773 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718777 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718779 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718782 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718785 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718789 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718793 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718795 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718798 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718801 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718803 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718805 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718808 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718810 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718812 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 07:48:17.722799 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718815 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718818 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718820 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718823 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718826 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718828 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718833 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718835 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718837 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718840 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718842 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718845 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718847 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718849 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718852 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718854 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718857 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718859 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718862 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718865 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 07:48:17.723301 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718867 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718870 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718872 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718876 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718879 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718882 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718884 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718887 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718889 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718892 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718894 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718897 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718899 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718902 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718904 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718907 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718909 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718911 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718915 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718918 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 07:48:17.723786 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718920 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718924 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718927 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718930 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718932 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718936 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718939 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718942 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718944 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718947 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718951 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718954 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718957 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718960 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.718963 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719035 2576 flags.go:64] FLAG: --address="0.0.0.0" Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719072 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719081 2576 flags.go:64] FLAG: --anonymous-auth="true" Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719085 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719089 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 05 07:48:17.724287 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719092 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719096 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719101 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719104 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719107 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719110 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719113 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719116 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719119 2576 flags.go:64] FLAG: --cgroup-root="" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719122 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719130 2576 flags.go:64] FLAG: --client-ca-file="" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719133 2576 flags.go:64] FLAG: --cloud-provider="external" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719136 2576 flags.go:64] FLAG: --cluster-dns="[]" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719140 2576 flags.go:64] FLAG: --cluster-domain="" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719143 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719146 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719149 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719152 2576 flags.go:64] FLAG: --container-log-max-files="5" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719156 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719159 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719162 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719165 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719169 2576 flags.go:64] FLAG: --contention-profiling="false" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719172 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 05 07:48:17.724774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719175 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719178 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719180 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719184 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719187 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719190 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719193 2576 flags.go:64] FLAG: --enable-load-reader="false" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719196 2576 flags.go:64] FLAG: --enable-server="true" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719199 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719204 2576 flags.go:64] FLAG: --event-burst="100" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719223 2576 flags.go:64] FLAG: --event-qps="50" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719227 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719232 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719236 2576 flags.go:64] FLAG: --eviction-hard="" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719240 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719243 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719246 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719248 2576 flags.go:64] FLAG: --eviction-soft="" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719253 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719255 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719260 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719262 2576 flags.go:64] FLAG: --experimental-mounter-path="" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719265 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719268 2576 flags.go:64] FLAG: --fail-swap-on="true" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719270 2576 flags.go:64] FLAG: --feature-gates="" Sep 05 07:48:17.725363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719274 2576 flags.go:64] FLAG: --file-check-frequency="20s" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719277 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719280 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719283 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719285 2576 flags.go:64] FLAG: --healthz-port="10248" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719289 2576 flags.go:64] FLAG: --help="false" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719292 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-143-184.ec2.internal" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719295 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719298 2576 flags.go:64] FLAG: --http-check-frequency="20s" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719300 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719304 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719307 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719310 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719313 2576 flags.go:64] FLAG: --image-service-endpoint="" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719316 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719319 2576 flags.go:64] FLAG: --kube-api-burst="100" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719321 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719324 2576 flags.go:64] FLAG: --kube-api-qps="50" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719327 2576 flags.go:64] FLAG: --kube-reserved="" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719330 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719332 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719335 2576 flags.go:64] FLAG: --kubelet-cgroups="" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719337 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719340 2576 flags.go:64] FLAG: --lock-file="" Sep 05 07:48:17.726039 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719343 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719347 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719350 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719359 2576 flags.go:64] FLAG: --log-json-split-stream="false" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719363 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719366 2576 flags.go:64] FLAG: --log-text-split-stream="false" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719369 2576 flags.go:64] FLAG: --logging-format="text" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719372 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719375 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719377 2576 flags.go:64] FLAG: --manifest-url="" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719380 2576 flags.go:64] FLAG: --manifest-url-header="" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719384 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719387 2576 flags.go:64] FLAG: --max-open-files="1000000" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719393 2576 flags.go:64] FLAG: --max-pods="110" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719395 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719398 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719401 2576 flags.go:64] FLAG: --memory-manager-policy="None" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719404 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719407 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719409 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719412 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719419 2576 flags.go:64] FLAG: --node-status-max-images="50" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719422 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719425 2576 flags.go:64] FLAG: --oom-score-adj="-999" Sep 05 07:48:17.726640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719428 2576 flags.go:64] FLAG: --pod-cidr="" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719431 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719436 2576 flags.go:64] FLAG: --pod-manifest-path="" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719439 2576 flags.go:64] FLAG: --pod-max-pids="-1" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719442 2576 flags.go:64] FLAG: --pods-per-core="0" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719444 2576 flags.go:64] FLAG: --port="10250" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719448 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719450 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0804992174a16a7ff" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719454 2576 flags.go:64] FLAG: --qos-reserved="" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719458 2576 flags.go:64] FLAG: --read-only-port="10255" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719460 2576 flags.go:64] FLAG: --register-node="true" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719463 2576 flags.go:64] FLAG: --register-with-taints="" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719467 2576 flags.go:64] FLAG: --registry-burst="10" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719470 2576 flags.go:64] FLAG: --registry-qps="5" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719473 2576 flags.go:64] FLAG: --reserved-cpus="" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719476 2576 flags.go:64] FLAG: --reserved-memory="" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719479 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719482 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719485 2576 flags.go:64] FLAG: --rotate-certificates="false" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719488 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719490 2576 flags.go:64] FLAG: --runonce="false" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719493 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719497 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719500 2576 flags.go:64] FLAG: --seccomp-default="false" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719503 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 05 07:48:17.727247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719506 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719509 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719511 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719514 2576 flags.go:64] FLAG: --storage-driver-password="root" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719517 2576 flags.go:64] FLAG: --storage-driver-secure="false" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719520 2576 flags.go:64] FLAG: --storage-driver-table="stats" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719523 2576 flags.go:64] FLAG: --storage-driver-user="root" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719526 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719528 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719531 2576 flags.go:64] FLAG: --system-cgroups="" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719534 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719539 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719541 2576 flags.go:64] FLAG: --tls-cert-file="" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719544 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719549 2576 flags.go:64] FLAG: --tls-min-version="" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719551 2576 flags.go:64] FLAG: --tls-private-key-file="" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719555 2576 flags.go:64] FLAG: --topology-manager-policy="none" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719558 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719561 2576 flags.go:64] FLAG: --topology-manager-scope="container" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719564 2576 flags.go:64] FLAG: --v="2" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719568 2576 flags.go:64] FLAG: --version="false" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719573 2576 flags.go:64] FLAG: --vmodule="" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719576 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719579 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719696 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 07:48:17.727863 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719701 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719704 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719707 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719710 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719714 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719716 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719719 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719722 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719724 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719727 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719730 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719732 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719735 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719737 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719740 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719742 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719745 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719747 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719750 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719753 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 07:48:17.728480 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719755 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719759 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719763 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719766 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719769 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719771 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719774 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719777 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719779 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719782 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719784 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719787 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719789 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719791 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719795 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719798 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719803 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719805 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719808 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 07:48:17.728963 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719811 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719814 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719816 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719819 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719821 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719824 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719826 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719829 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719832 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719834 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719837 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719840 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719842 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719845 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719848 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719852 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719855 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719857 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719860 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719862 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 07:48:17.729593 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719865 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719867 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719870 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719874 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719876 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719878 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719881 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719883 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719886 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719890 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719892 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719894 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719897 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719899 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719902 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719905 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719907 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719909 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719912 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719915 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 07:48:17.730071 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719917 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719919 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719922 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719924 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719927 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719929 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719932 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719935 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719937 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719939 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719942 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719945 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719947 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719950 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719952 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719954 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719957 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719960 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 07:48:17.730566 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.719962 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.719968 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.727822 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.727839 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.727855 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.727861 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727918 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727923 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727926 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727929 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727932 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727935 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727937 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727940 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727942 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 07:48:17.730996 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727945 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727947 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727950 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727952 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727955 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727958 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727960 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727963 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727965 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727968 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727971 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727973 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727976 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727979 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727981 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727984 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727987 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727989 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727992 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 07:48:17.731393 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727994 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.727998 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728001 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728003 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728006 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728009 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728011 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728014 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728016 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728019 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728021 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728024 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728026 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728029 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728031 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728034 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728036 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728039 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728041 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728044 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 07:48:17.731858 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728047 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728049 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728053 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728057 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728059 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728063 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728065 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728068 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728071 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728074 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728077 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728081 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728083 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728086 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728090 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728093 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728095 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728098 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728100 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 07:48:17.732370 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728103 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728106 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728108 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728111 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728113 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728115 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728118 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728121 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728123 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728126 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728128 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728131 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728133 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728136 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728138 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728141 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728143 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728147 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728151 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728154 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 07:48:17.732855 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728157 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728159 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728162 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728165 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728167 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728170 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728173 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728175 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728178 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728181 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728183 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728186 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.728191 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728317 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728322 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 05 07:48:17.733351 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728325 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728327 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728330 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728333 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728335 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728338 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728341 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728343 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728345 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728348 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728350 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728353 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728355 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728358 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728360 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728363 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728366 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728370 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728373 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 05 07:48:17.733730 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728377 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728381 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728383 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728386 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728389 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728392 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728395 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728398 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728400 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728403 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728406 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728409 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728413 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728415 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728418 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728420 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728423 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728425 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728428 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 05 07:48:17.734192 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728430 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728433 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728435 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728438 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728440 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728443 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728445 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728448 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728450 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728453 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728455 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728458 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728461 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728463 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728465 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728468 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728470 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728473 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728476 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728478 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 05 07:48:17.734672 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728481 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728484 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728487 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728489 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728492 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728495 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728497 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728500 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728503 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728505 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728507 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728510 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728513 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728515 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728517 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728520 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728522 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728525 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728527 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728529 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 05 07:48:17.735152 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728532 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728535 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728537 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728540 2576 feature_gate.go:328] unrecognized feature gate: Example Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728542 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728545 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728547 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728550 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728552 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728555 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728557 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728560 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728563 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728565 2576 feature_gate.go:328] unrecognized feature gate: Example2 Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728568 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728571 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728573 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728575 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 05 07:48:17.735648 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:17.728578 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 05 07:48:17.736088 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.728583 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 05 07:48:17.736088 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.728693 2576 server.go:962] "Client rotation is on, will bootstrap in background" Sep 05 07:48:17.736088 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.731201 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 05 07:48:17.736088 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.732304 2576 server.go:1019] "Starting client certificate rotation" Sep 05 07:48:17.736088 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.732399 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 05 07:48:17.736088 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.732434 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 05 07:48:17.736088 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.735145 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 05 07:48:17.736687 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.736668 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 05 07:48:17.748310 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.748292 2576 log.go:25] "Validated CRI v1 runtime API" Sep 05 07:48:17.753524 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.753502 2576 log.go:25] "Validated CRI v1 image API" Sep 05 07:48:17.755955 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.755933 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 05 07:48:17.759447 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.759424 2576 fs.go:135] Filesystem UUIDs: map[56e142c4-a9ec-4666-8cee-a81d13a70faf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fde23d97-4ec4-4bc6-9428-5bacb7be5a4a:/dev/nvme0n1p3] Sep 05 07:48:17.759561 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.759446 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 05 07:48:17.763749 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.763621 2576 manager.go:216] Machine: {Timestamp:2026-09-05 07:48:17.763121148 +0000 UTC m=+0.372096262 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105815 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec201caefbec2311c7447b2e7f3f4616 SystemUUID:ec201cae-fbec-2311-c744-7b2e7f3f4616 BootID:c8b94e15-6d12-4490-9d49-febb1b01b44b Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:23:c2:da:e7:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:23:c2:da:e7:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:57:28:91:16:be Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 05 07:48:17.763749 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.763732 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 05 07:48:17.763928 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.763884 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 05 07:48:17.766182 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.766163 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 05 07:48:17.767444 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.767414 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 05 07:48:17.767614 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.767451 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-184.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 05 07:48:17.769100 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.769088 2576 topology_manager.go:138] "Creating topology manager with none policy" Sep 05 07:48:17.769156 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.769107 2576 container_manager_linux.go:309] "Creating device plugin manager" Sep 05 07:48:17.769156 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.769119 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 05 07:48:17.770103 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.770091 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 05 07:48:17.772145 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.772132 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 05 07:48:17.777666 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.777652 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 05 07:48:17.777803 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.777792 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 05 07:48:17.777868 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.777831 2576 kubelet.go:486] "Attempting to sync node with API server" Sep 05 07:48:17.777868 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.777846 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 05 07:48:17.777960 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.777868 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 05 07:48:17.777960 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.777887 2576 kubelet.go:398] "Adding apiserver pod source" Sep 05 07:48:17.777960 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.777901 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 05 07:48:17.778309 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.778295 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 05 07:48:17.778390 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.778319 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 05 07:48:17.780698 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.780681 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 05 07:48:17.783934 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.783918 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 05 07:48:17.784005 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.783950 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 05 07:48:17.784117 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.784106 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 05 07:48:17.784160 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.784121 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 05 07:48:17.784160 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.784128 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 05 07:48:17.784160 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.784133 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 05 07:48:17.784160 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.784141 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 05 07:48:17.784160 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.784146 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 05 07:48:17.784726 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.784716 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 05 07:48:17.784726 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.784726 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 05 07:48:17.784778 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.784733 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 05 07:48:17.784778 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.784738 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 05 07:48:17.784778 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.784746 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 05 07:48:17.784778 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.784753 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 05 07:48:17.785648 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.785639 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 05 07:48:17.785648 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.785648 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 05 07:48:17.790621 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.790602 2576 server.go:1268] "Started kubelet" Sep 05 07:48:17.790708 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.790661 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 05 07:48:17.790768 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.790728 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 05 07:48:17.790814 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.790797 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 05 07:48:17.791471 ip-10-0-143-184 systemd[1]: Started Kubernetes Kubelet. Sep 05 07:48:17.792556 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.792446 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 05 07:48:17.793045 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.792981 2576 server.go:310] "Adding debug handlers to kubelet server" Sep 05 07:48:17.793190 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.793171 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-184.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 05 07:48:17.793271 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.793241 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 05 07:48:17.793536 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.793512 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-184.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 05 07:48:17.795268 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.795250 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 05 07:48:17.797513 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.796414 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-184.ec2.internal.18d25d3828065388 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-184.ec2.internal,UID:ip-10-0-143-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-184.ec2.internal,},FirstTimestamp:2026-09-05 07:48:17.790563208 +0000 UTC m=+0.399538322,LastTimestamp:2026-09-05 07:48:17.790563208 +0000 UTC m=+0.399538322,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-184.ec2.internal,}" Sep 05 07:48:17.798499 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.798486 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 05 07:48:17.798548 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.798498 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 05 07:48:17.798875 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.798685 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:17.799234 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.799193 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 05 07:48:17.799234 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.799193 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 05 07:48:17.799422 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.799239 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 05 07:48:17.799422 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.799294 2576 reconstruct.go:97] "Volume reconstruction finished" Sep 05 07:48:17.799422 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.799301 2576 reconciler.go:29] "Reconciler: start to sync state" Sep 05 07:48:17.799607 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.799593 2576 factory.go:153] Registering CRI-O factory Sep 05 07:48:17.799648 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.799637 2576 factory.go:223] Registration of the crio container factory successfully Sep 05 07:48:17.799704 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.799694 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 05 07:48:17.799740 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.799705 2576 factory.go:55] Registering systemd factory Sep 05 07:48:17.799740 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.799711 2576 factory.go:223] Registration of the systemd container factory successfully Sep 05 07:48:17.799740 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.799722 2576 factory.go:103] Registering Raw factory Sep 05 07:48:17.799740 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.799731 2576 manager.go:1196] Started watching for new ooms in manager Sep 05 07:48:17.800110 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.800099 2576 manager.go:349] Starting recovery of all containers Sep 05 07:48:17.806141 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.806113 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-184.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 05 07:48:17.806306 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.806279 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 05 07:48:17.810280 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.808388 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 05 07:48:17.813273 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.813256 2576 manager.go:354] Recovery completed Sep 05 07:48:17.817582 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.817565 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 07:48:17.819422 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.819397 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientMemory" Sep 05 07:48:17.819492 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.819433 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 07:48:17.819492 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.819444 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientPID" Sep 05 07:48:17.819989 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.819975 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 05 07:48:17.819989 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.819988 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 05 07:48:17.820066 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.820004 2576 state_mem.go:36] "Initialized new in-memory state store" Sep 05 07:48:17.820066 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.820010 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 07:48:17.821578 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.821564 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientMemory" Sep 05 07:48:17.821624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.821591 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 07:48:17.821624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.821601 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientPID" Sep 05 07:48:17.822187 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.822175 2576 policy_none.go:49] "None policy: Start" Sep 05 07:48:17.822240 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.822193 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 05 07:48:17.822240 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.822205 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 05 07:48:17.824147 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.824137 2576 policy_none.go:47] "Start" Sep 05 07:48:17.829880 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.829808 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-184.ec2.internal.18d25d3829bea51e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-184.ec2.internal,UID:ip-10-0-143-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-184.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-184.ec2.internal,},FirstTimestamp:2026-09-05 07:48:17.819419934 +0000 UTC m=+0.428395047,LastTimestamp:2026-09-05 07:48:17.819419934 +0000 UTC m=+0.428395047,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-184.ec2.internal,}" Sep 05 07:48:17.850667 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.850599 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-184.ec2.internal.18d25d3829beec78 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-184.ec2.internal,UID:ip-10-0-143-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-184.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-184.ec2.internal,},FirstTimestamp:2026-09-05 07:48:17.8194382 +0000 UTC m=+0.428413314,LastTimestamp:2026-09-05 07:48:17.8194382 +0000 UTC m=+0.428413314,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-184.ec2.internal,}" Sep 05 07:48:17.864345 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.864277 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-184.ec2.internal.18d25d3829bf10c0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-184.ec2.internal,UID:ip-10-0-143-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-184.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-184.ec2.internal,},FirstTimestamp:2026-09-05 07:48:17.819447488 +0000 UTC m=+0.428422601,LastTimestamp:2026-09-05 07:48:17.819447488 +0000 UTC m=+0.428422601,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-184.ec2.internal,}" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.870808 2576 manager.go:337] "Starting Device Plugin manager" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.870844 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.870857 2576 server.go:87] "Starting device plugin registration server" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.871155 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.871171 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.871302 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.871374 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.871382 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.871906 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.871958 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.876668 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-184.ec2.internal.18d25d3829bea51e\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-184.ec2.internal.18d25d3829bea51e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-184.ec2.internal,UID:ip-10-0-143-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-184.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-184.ec2.internal,},FirstTimestamp:2026-09-05 07:48:17.819419934 +0000 UTC m=+0.428395047,LastTimestamp:2026-09-05 07:48:17.821578245 +0000 UTC m=+0.430553359,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-184.ec2.internal,}" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.879758 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.879799 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.879817 2576 kubelet.go:2448] "Starting kubelet main sync loop" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.879851 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.879878 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nlrx7" Sep 05 07:48:17.887825 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.886896 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nlrx7" Sep 05 07:48:17.888372 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.887928 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-184.ec2.internal.18d25d3829beec78\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-184.ec2.internal.18d25d3829beec78 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-184.ec2.internal,UID:ip-10-0-143-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-184.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-184.ec2.internal,},FirstTimestamp:2026-09-05 07:48:17.8194382 +0000 UTC m=+0.428413314,LastTimestamp:2026-09-05 07:48:17.821595838 +0000 UTC m=+0.430570952,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-184.ec2.internal,}" Sep 05 07:48:17.888372 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.888006 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" Sep 05 07:48:17.889854 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.889783 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-184.ec2.internal.18d25d3829bf10c0\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-184.ec2.internal.18d25d3829bf10c0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-184.ec2.internal,UID:ip-10-0-143-184.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-184.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-184.ec2.internal,},FirstTimestamp:2026-09-05 07:48:17.819447488 +0000 UTC m=+0.428422601,LastTimestamp:2026-09-05 07:48:17.821604747 +0000 UTC m=+0.430579862,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-184.ec2.internal,}" Sep 05 07:48:17.971826 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.971709 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 07:48:17.972762 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.972744 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientMemory" Sep 05 07:48:17.972888 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.972778 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 07:48:17.972888 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.972789 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientPID" Sep 05 07:48:17.972888 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.972811 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-184.ec2.internal" Sep 05 07:48:17.980897 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.980866 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-184.ec2.internal"] Sep 05 07:48:17.980995 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.980940 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 07:48:17.982781 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.982761 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientMemory" Sep 05 07:48:17.982870 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.982788 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 07:48:17.982870 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.982798 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientPID" Sep 05 07:48:17.983405 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.983387 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-184.ec2.internal" Sep 05 07:48:17.983475 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:17.983413 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-184.ec2.internal\": node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:17.984974 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.984962 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 07:48:17.985096 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.985070 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal" Sep 05 07:48:17.985145 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.985112 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 07:48:17.985745 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.985729 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientMemory" Sep 05 07:48:17.985835 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.985753 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 07:48:17.985835 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.985734 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientMemory" Sep 05 07:48:17.985835 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.985762 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientPID" Sep 05 07:48:17.985835 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.985774 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 07:48:17.985835 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.985784 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientPID" Sep 05 07:48:17.987872 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.987857 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-184.ec2.internal" Sep 05 07:48:17.987916 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.987884 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 07:48:17.994274 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.994257 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientMemory" Sep 05 07:48:17.994364 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.994285 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 07:48:17.994364 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:17.994297 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientPID" Sep 05 07:48:18.006832 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:18.006813 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-184.ec2.internal\" not found" node="ip-10-0-143-184.ec2.internal" Sep 05 07:48:18.008733 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:18.008717 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:18.013155 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:18.013136 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-184.ec2.internal\" not found" node="ip-10-0-143-184.ec2.internal" Sep 05 07:48:18.100725 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.100678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00a91b5c1abc892797819c828b866aed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal\" (UID: \"00a91b5c1abc892797819c828b866aed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal" Sep 05 07:48:18.100725 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.100728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00a91b5c1abc892797819c828b866aed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal\" (UID: \"00a91b5c1abc892797819c828b866aed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal" Sep 05 07:48:18.100943 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.100754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76bb8a07519e0ff54e97f8d73d9e9f48-config\") pod \"kube-apiserver-proxy-ip-10-0-143-184.ec2.internal\" (UID: \"76bb8a07519e0ff54e97f8d73d9e9f48\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-184.ec2.internal" Sep 05 07:48:18.109469 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:18.109442 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:18.201315 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.201276 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00a91b5c1abc892797819c828b866aed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal\" (UID: \"00a91b5c1abc892797819c828b866aed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal" Sep 05 07:48:18.201504 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.201329 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00a91b5c1abc892797819c828b866aed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal\" (UID: \"00a91b5c1abc892797819c828b866aed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal" Sep 05 07:48:18.201504 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.201348 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76bb8a07519e0ff54e97f8d73d9e9f48-config\") pod \"kube-apiserver-proxy-ip-10-0-143-184.ec2.internal\" (UID: \"76bb8a07519e0ff54e97f8d73d9e9f48\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-184.ec2.internal" Sep 05 07:48:18.201504 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.201290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/00a91b5c1abc892797819c828b866aed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal\" (UID: \"00a91b5c1abc892797819c828b866aed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal" Sep 05 07:48:18.201504 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.201384 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76bb8a07519e0ff54e97f8d73d9e9f48-config\") pod \"kube-apiserver-proxy-ip-10-0-143-184.ec2.internal\" (UID: \"76bb8a07519e0ff54e97f8d73d9e9f48\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-184.ec2.internal" Sep 05 07:48:18.201504 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.201421 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/00a91b5c1abc892797819c828b866aed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal\" (UID: \"00a91b5c1abc892797819c828b866aed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal" Sep 05 07:48:18.210324 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:18.210303 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:18.311194 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:18.311120 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:18.311194 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.311121 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal" Sep 05 07:48:18.317516 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.317499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-184.ec2.internal" Sep 05 07:48:18.412186 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:18.412136 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:18.512664 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:18.512628 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:18.613323 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:18.613237 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:18.662457 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.662427 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 07:48:18.714150 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:18.714118 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:18.721125 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.721103 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 07:48:18.732443 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.732421 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 05 07:48:18.732570 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.732550 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 05 07:48:18.732602 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.732551 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 05 07:48:18.814498 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:18.814468 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:18.856634 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.856609 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 05 07:48:18.857910 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.857892 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientMemory" Sep 05 07:48:18.858015 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.857922 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasNoDiskPressure" Sep 05 07:48:18.858015 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.857931 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeHasSufficientPID" Sep 05 07:48:18.889624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.889584 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-04 07:43:18 +0000 UTC" deadline="2028-05-04 04:18:36.98531877 +0000 UTC" Sep 05 07:48:18.889624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:18.889615 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14564h30m18.095708443s" Sep 05 07:48:18.915229 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:18.915040 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:18.999967 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:18.999936 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a91b5c1abc892797819c828b866aed.slice/crio-6e1638acbe21d01ef811741822868e51d7f0a1a9c333a77df2e8f058a3281954 WatchSource:0}: Error finding container 6e1638acbe21d01ef811741822868e51d7f0a1a9c333a77df2e8f058a3281954: Status 404 returned error can't find the container with id 6e1638acbe21d01ef811741822868e51d7f0a1a9c333a77df2e8f058a3281954 Sep 05 07:48:19.000490 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:19.000468 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bb8a07519e0ff54e97f8d73d9e9f48.slice/crio-3fdccbb4096948ec1c0a69b1ea24fe398588bc7666134e1327383f2ad454b4b0 WatchSource:0}: Error finding container 3fdccbb4096948ec1c0a69b1ea24fe398588bc7666134e1327383f2ad454b4b0: Status 404 returned error can't find the container with id 3fdccbb4096948ec1c0a69b1ea24fe398588bc7666134e1327383f2ad454b4b0 Sep 05 07:48:19.007200 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.007178 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 07:48:19.015621 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:19.015603 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:19.116434 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:19.116345 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-184.ec2.internal\" not found" Sep 05 07:48:19.134608 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.134583 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 07:48:19.167368 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.167342 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 07:48:19.198966 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.198939 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal" Sep 05 07:48:19.210323 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.210298 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 05 07:48:19.211529 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.211512 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-184.ec2.internal" Sep 05 07:48:19.220397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.220373 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 05 07:48:19.779875 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.779572 2576 apiserver.go:52] "Watching apiserver" Sep 05 07:48:19.786673 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.786466 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 05 07:48:19.790034 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.789995 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-zbn86","kube-system/kube-apiserver-proxy-ip-10-0-143-184.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d","openshift-cluster-node-tuning-operator/tuned-vhlrm","openshift-dns/node-resolver-4bmbd","openshift-image-registry/node-ca-brf5d","openshift-multus/multus-additional-cni-plugins-dsknf","openshift-multus/network-metrics-daemon-zr6kd","openshift-network-operator/iptables-alerter-rhbv5","kube-system/global-pull-secret-syncer-xtq7n","kube-system/konnectivity-agent-xpmql","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal","openshift-multus/multus-l224c","openshift-network-diagnostics/network-check-target-dvr7g"] Sep 05 07:48:19.793201 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.793179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.796509 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.795368 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l224c" Sep 05 07:48:19.796509 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.796308 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 05 07:48:19.796509 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.796387 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 05 07:48:19.796509 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.796436 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 05 07:48:19.796509 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.796442 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 05 07:48:19.796826 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.796815 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-j5b74\"" Sep 05 07:48:19.797064 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.797047 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 05 07:48:19.797776 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.797751 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 05 07:48:19.798273 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.798251 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rtrjr\"" Sep 05 07:48:19.798726 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.798705 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.798856 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.798838 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 05 07:48:19.801790 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.801769 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 05 07:48:19.802124 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.802106 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 05 07:48:19.802372 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.802357 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s45kq\"" Sep 05 07:48:19.802632 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.802617 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 05 07:48:19.806252 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.804871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.807405 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.806965 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4bmbd" Sep 05 07:48:19.807502 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.807477 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 05 07:48:19.807875 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.807852 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 05 07:48:19.808675 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.808652 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l7hb5\"" Sep 05 07:48:19.809724 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809439 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-brf5d" Sep 05 07:48:19.809724 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.809724 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-system-cni-dir\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.809724 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-multus-socket-dir-parent\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.809724 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-var-lib-cni-bin\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.809724 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-var-lib-cni-multus\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69l4q\" (UniqueName: \"kubernetes.io/projected/f7fd8127-7a31-414b-962a-84b1e0303114-kube-api-access-69l4q\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-cnibin\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-run-k8s-cni-cncf-io\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvnr6\" (UniqueName: \"kubernetes.io/projected/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-kube-api-access-nvnr6\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809875 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-sys-fs\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809899 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-sysconfig\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-systemd\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809948 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-system-cni-dir\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.809972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-cnibin\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-multus-daemon-config\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810029 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-etc-kubernetes\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810057 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.810079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-cni-binary-copy\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-hostroot\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-multus-conf-dir\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-run-multus-certs\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810191 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-host\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810216 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-socket-dir\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-modprobe-d\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810269 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-sysctl-d\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810313 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-sysctl-conf\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810336 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-run\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-etc-selinux\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810422 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-lib-modules\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-var-lib-kubelet\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810472 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-tuned\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2848\" (UniqueName: \"kubernetes.io/projected/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-kube-api-access-h2848\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810524 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtnmz\" (UniqueName: \"kubernetes.io/projected/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-kube-api-access-gtnmz\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.810798 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-var-lib-kubelet\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.811599 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810578 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-registration-dir\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.811599 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-sys\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.811599 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.811599 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-cni-binary-copy\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.811599 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-multus-cni-dir\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.811599 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-run-netns\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.811599 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-device-dir\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.811599 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810766 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-kubernetes\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.811599 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.810790 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-tmp\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.812088 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.811826 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 05 07:48:19.812088 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.812018 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 05 07:48:19.812192 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.812183 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7mc4j\"" Sep 05 07:48:19.812917 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.812896 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 05 07:48:19.813213 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.813196 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 05 07:48:19.813993 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.813629 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rhbv5" Sep 05 07:48:19.813993 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.813708 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vrbgv\"" Sep 05 07:48:19.814973 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.814295 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 05 07:48:19.814973 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.814586 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 05 07:48:19.816184 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.816154 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 05 07:48:19.816906 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.816884 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fc4kx\"" Sep 05 07:48:19.817281 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.817262 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 05 07:48:19.817752 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.817641 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 05 07:48:19.817752 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.817691 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.819189 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.819162 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:19.819325 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:19.819303 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:19.821952 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.821927 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:19.822050 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:19.822005 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:19.823995 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.823978 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 05 07:48:19.826040 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.826005 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xpmql" Sep 05 07:48:19.836235 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.830234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:19.836235 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:19.830319 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:19.836235 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.832097 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 05 07:48:19.836235 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.832145 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 05 07:48:19.836235 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.832165 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bxzsd\"" Sep 05 07:48:19.836235 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.832099 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 05 07:48:19.836235 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.832440 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-crlpv\"" Sep 05 07:48:19.836235 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.832570 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 05 07:48:19.836235 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.832618 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 05 07:48:19.836235 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.832659 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 05 07:48:19.836235 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.832570 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 05 07:48:19.890106 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.890044 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-184.ec2.internal" event={"ID":"76bb8a07519e0ff54e97f8d73d9e9f48","Type":"ContainerStarted","Data":"3fdccbb4096948ec1c0a69b1ea24fe398588bc7666134e1327383f2ad454b4b0"} Sep 05 07:48:19.894038 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.894008 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal" event={"ID":"00a91b5c1abc892797819c828b866aed","Type":"ContainerStarted","Data":"6e1638acbe21d01ef811741822868e51d7f0a1a9c333a77df2e8f058a3281954"} Sep 05 07:48:19.901582 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.901560 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 05 07:48:19.911771 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.911739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-system-cni-dir\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.911909 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.911814 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-cnibin\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.911909 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.911854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-etc-kubernetes\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.912007 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.911918 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13432308-4a00-44e2-babf-6daaff232a0a-serviceca\") pod \"node-ca-brf5d\" (UID: \"13432308-4a00-44e2-babf-6daaff232a0a\") " pod="openshift-image-registry/node-ca-brf5d" Sep 05 07:48:19.912007 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.911963 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-etc-kubernetes\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.912007 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.911979 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-run-netns\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.912176 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912023 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-system-cni-dir\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.912176 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.911989 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-cnibin\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.912574 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.912662 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912605 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-hostroot\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.912662 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912637 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-run-multus-certs\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.912763 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-host\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.912763 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-sysctl-d\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.912763 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912715 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-run\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.912763 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912741 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lz68\" (UniqueName: \"kubernetes.io/projected/13432308-4a00-44e2-babf-6daaff232a0a-kube-api-access-7lz68\") pod \"node-ca-brf5d\" (UID: \"13432308-4a00-44e2-babf-6daaff232a0a\") " pod="openshift-image-registry/node-ca-brf5d" Sep 05 07:48:19.912948 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4702c99f-9747-4b1d-853d-090e6a7327ba-host-slash\") pod \"iptables-alerter-rhbv5\" (UID: \"4702c99f-9747-4b1d-853d-090e6a7327ba\") " pod="openshift-network-operator/iptables-alerter-rhbv5" Sep 05 07:48:19.912948 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-cni-netd\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.912948 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-tuned\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.912948 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-slash\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.912948 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912866 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-run-openvswitch\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.912948 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912893 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtnmz\" (UniqueName: \"kubernetes.io/projected/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-kube-api-access-gtnmz\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.912948 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-var-lib-kubelet\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.912948 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-registration-dir\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.912948 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-sys\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-ovn-node-metrics-cert\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.912996 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-ovnkube-script-lib\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svxtm\" (UniqueName: \"kubernetes.io/projected/a6bf702d-539a-4164-9e92-18a93205e30f-kube-api-access-svxtm\") pod \"node-resolver-4bmbd\" (UID: \"a6bf702d-539a-4164-9e92-18a93205e30f\") " pod="openshift-dns/node-resolver-4bmbd" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b43f76a2-cd92-49a6-b567-71f3ad019d30-dbus\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913093 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-cni-binary-copy\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-multus-cni-dir\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-run-netns\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-kubernetes\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-var-lib-openvswitch\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs\") pod \"network-metrics-daemon-zr6kd\" (UID: \"73836491-7624-4e24-a8cd-2ad60bc09155\") " pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913227 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-system-cni-dir\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913332 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69l4q\" (UniqueName: \"kubernetes.io/projected/f7fd8127-7a31-414b-962a-84b1e0303114-kube-api-access-69l4q\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913354 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-registration-dir\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvnr6\" (UniqueName: \"kubernetes.io/projected/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-kube-api-access-nvnr6\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.913401 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913366 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-hostroot\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913424 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-host\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913428 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-sys-fs\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-sysconfig\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913462 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-var-lib-kubelet\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913483 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-systemd\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913497 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-run\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-ovnkube-config\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913537 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-multus-daemon-config\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913553 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-multus-cni-dir\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913564 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-kubelet\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913584 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-run-netns\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m4tw\" (UniqueName: \"kubernetes.io/projected/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-kube-api-access-7m4tw\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913581 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913620 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-run-multus-certs\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913683 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913751 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-system-cni-dir\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913835 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-sysctl-d\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.913921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913879 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-kubernetes\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913926 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-sys\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.913966 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b43f76a2-cd92-49a6-b567-71f3ad019d30-kubelet-config\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-cni-binary-copy\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914029 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-multus-conf-dir\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914060 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2hxx\" (UniqueName: \"kubernetes.io/projected/4702c99f-9747-4b1d-853d-090e6a7327ba-kube-api-access-g2hxx\") pod \"iptables-alerter-rhbv5\" (UID: \"4702c99f-9747-4b1d-853d-090e6a7327ba\") " pod="openshift-network-operator/iptables-alerter-rhbv5" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dea8be0d-b3bc-4bb9-80d5-00bf1d08a034-agent-certs\") pod \"konnectivity-agent-xpmql\" (UID: \"dea8be0d-b3bc-4bb9-80d5-00bf1d08a034\") " pod="kube-system/konnectivity-agent-xpmql" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-socket-dir\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914120 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-cni-binary-copy\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-modprobe-d\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-sysctl-conf\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4702c99f-9747-4b1d-853d-090e6a7327ba-iptables-alerter-script\") pod \"iptables-alerter-rhbv5\" (UID: \"4702c99f-9747-4b1d-853d-090e6a7327ba\") " pod="openshift-network-operator/iptables-alerter-rhbv5" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914212 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914228 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-run-ovn\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914257 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-log-socket\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-socket-dir\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.914732 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914253 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-multus-daemon-config\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914311 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-modprobe-d\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-cni-bin\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-etc-selinux\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914353 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-multus-conf-dir\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-lib-modules\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914439 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-sysctl-conf\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914467 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-etc-selinux\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914469 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-lib-modules\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914350 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-sysconfig\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-var-lib-kubelet\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-systemd\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2848\" (UniqueName: \"kubernetes.io/projected/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-kube-api-access-h2848\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13432308-4a00-44e2-babf-6daaff232a0a-host\") pod \"node-ca-brf5d\" (UID: \"13432308-4a00-44e2-babf-6daaff232a0a\") " pod="openshift-image-registry/node-ca-brf5d" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914650 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-etc-openvswitch\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914650 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-var-lib-kubelet\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914697 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.915397 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xt7z\" (UniqueName: \"kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z\") pod \"network-check-target-dvr7g\" (UID: \"dbfac494-fbe8-4979-ba88-fcf267911128\") " pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914733 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dea8be0d-b3bc-4bb9-80d5-00bf1d08a034-konnectivity-ca\") pod \"konnectivity-agent-xpmql\" (UID: \"dea8be0d-b3bc-4bb9-80d5-00bf1d08a034\") " pod="kube-system/konnectivity-agent-xpmql" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-device-dir\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914809 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-tmp\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-env-overrides\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914876 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-sys-fs\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914909 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f7fd8127-7a31-414b-962a-84b1e0303114-device-dir\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914931 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhnz\" (UniqueName: \"kubernetes.io/projected/73836491-7624-4e24-a8cd-2ad60bc09155-kube-api-access-wwhnz\") pod \"network-metrics-daemon-zr6kd\" (UID: \"73836491-7624-4e24-a8cd-2ad60bc09155\") " pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.914987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-multus-socket-dir-parent\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-var-lib-cni-bin\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915022 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-cni-binary-copy\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-var-lib-cni-multus\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915062 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-run-systemd\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-multus-socket-dir-parent\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-node-log\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.916188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6bf702d-539a-4164-9e92-18a93205e30f-tmp-dir\") pod \"node-resolver-4bmbd\" (UID: \"a6bf702d-539a-4164-9e92-18a93205e30f\") " pod="openshift-dns/node-resolver-4bmbd" Sep 05 07:48:19.917229 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915234 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-var-lib-cni-bin\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.917229 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915235 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-var-lib-cni-multus\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.917229 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-cnibin\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.917229 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915318 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-cnibin\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.917229 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.917229 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-run-k8s-cni-cncf-io\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.917229 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915561 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-host-run-k8s-cni-cncf-io\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.917229 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-systemd-units\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:19.917229 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6bf702d-539a-4164-9e92-18a93205e30f-hosts-file\") pod \"node-resolver-4bmbd\" (UID: \"a6bf702d-539a-4164-9e92-18a93205e30f\") " pod="openshift-dns/node-resolver-4bmbd" Sep 05 07:48:19.917229 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.915832 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.917229 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.916680 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvxs4" Sep 05 07:48:19.923245 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.919523 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-tmp\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.923245 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.920274 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-etc-tuned\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.925559 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.925530 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtnmz\" (UniqueName: \"kubernetes.io/projected/e1d05ed4-423c-4619-89c1-3d4a00b8ea86-kube-api-access-gtnmz\") pod \"multus-additional-cni-plugins-dsknf\" (UID: \"e1d05ed4-423c-4619-89c1-3d4a00b8ea86\") " pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:19.926100 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.926076 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vvxs4" Sep 05 07:48:19.927070 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.927043 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvnr6\" (UniqueName: \"kubernetes.io/projected/861a7bc4-b5ce-4432-9ba4-c2f938aab3f3-kube-api-access-nvnr6\") pod \"multus-l224c\" (UID: \"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3\") " pod="openshift-multus/multus-l224c" Sep 05 07:48:19.928494 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.927531 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2848\" (UniqueName: \"kubernetes.io/projected/b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd-kube-api-access-h2848\") pod \"tuned-vhlrm\" (UID: \"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd\") " pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:19.932200 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:19.932165 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69l4q\" (UniqueName: \"kubernetes.io/projected/f7fd8127-7a31-414b-962a-84b1e0303114-kube-api-access-69l4q\") pod \"aws-ebs-csi-driver-node-vck7d\" (UID: \"f7fd8127-7a31-414b-962a-84b1e0303114\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:20.017076 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.016767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-run-systemd\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017076 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.016799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-node-log\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017076 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.016817 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6bf702d-539a-4164-9e92-18a93205e30f-tmp-dir\") pod \"node-resolver-4bmbd\" (UID: \"a6bf702d-539a-4164-9e92-18a93205e30f\") " pod="openshift-dns/node-resolver-4bmbd" Sep 05 07:48:20.017076 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.016834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-systemd-units\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017076 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.016856 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6bf702d-539a-4164-9e92-18a93205e30f-hosts-file\") pod \"node-resolver-4bmbd\" (UID: \"a6bf702d-539a-4164-9e92-18a93205e30f\") " pod="openshift-dns/node-resolver-4bmbd" Sep 05 07:48:20.017076 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.016927 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a6bf702d-539a-4164-9e92-18a93205e30f-hosts-file\") pod \"node-resolver-4bmbd\" (UID: \"a6bf702d-539a-4164-9e92-18a93205e30f\") " pod="openshift-dns/node-resolver-4bmbd" Sep 05 07:48:20.017076 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.016928 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-node-log\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017076 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.016992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13432308-4a00-44e2-babf-6daaff232a0a-serviceca\") pod \"node-ca-brf5d\" (UID: \"13432308-4a00-44e2-babf-6daaff232a0a\") " pod="openshift-image-registry/node-ca-brf5d" Sep 05 07:48:20.017076 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017000 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-systemd-units\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017076 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-run-netns\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017076 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017048 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-run-systemd\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lz68\" (UniqueName: \"kubernetes.io/projected/13432308-4a00-44e2-babf-6daaff232a0a-kube-api-access-7lz68\") pod \"node-ca-brf5d\" (UID: \"13432308-4a00-44e2-babf-6daaff232a0a\") " pod="openshift-image-registry/node-ca-brf5d" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4702c99f-9747-4b1d-853d-090e6a7327ba-host-slash\") pod \"iptables-alerter-rhbv5\" (UID: \"4702c99f-9747-4b1d-853d-090e6a7327ba\") " pod="openshift-network-operator/iptables-alerter-rhbv5" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-cni-netd\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017202 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-slash\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017219 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6bf702d-539a-4164-9e92-18a93205e30f-tmp-dir\") pod \"node-resolver-4bmbd\" (UID: \"a6bf702d-539a-4164-9e92-18a93205e30f\") " pod="openshift-dns/node-resolver-4bmbd" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017134 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-run-netns\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-run-openvswitch\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-ovn-node-metrics-cert\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-ovnkube-script-lib\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svxtm\" (UniqueName: \"kubernetes.io/projected/a6bf702d-539a-4164-9e92-18a93205e30f-kube-api-access-svxtm\") pod \"node-resolver-4bmbd\" (UID: \"a6bf702d-539a-4164-9e92-18a93205e30f\") " pod="openshift-dns/node-resolver-4bmbd" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017370 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b43f76a2-cd92-49a6-b567-71f3ad019d30-dbus\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017406 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-var-lib-openvswitch\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs\") pod \"network-metrics-daemon-zr6kd\" (UID: \"73836491-7624-4e24-a8cd-2ad60bc09155\") " pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017472 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-ovnkube-config\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-kubelet\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017530 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m4tw\" (UniqueName: \"kubernetes.io/projected/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-kube-api-access-7m4tw\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.017631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017533 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4702c99f-9747-4b1d-853d-090e6a7327ba-host-slash\") pod \"iptables-alerter-rhbv5\" (UID: \"4702c99f-9747-4b1d-853d-090e6a7327ba\") " pod="openshift-network-operator/iptables-alerter-rhbv5" Sep 05 07:48:20.018230 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017556 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13432308-4a00-44e2-babf-6daaff232a0a-serviceca\") pod \"node-ca-brf5d\" (UID: \"13432308-4a00-44e2-babf-6daaff232a0a\") " pod="openshift-image-registry/node-ca-brf5d" Sep 05 07:48:20.018230 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017716 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b43f76a2-cd92-49a6-b567-71f3ad019d30-dbus\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:20.018230 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-cni-netd\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018230 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017775 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-kubelet\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018230 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.017840 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-var-lib-openvswitch\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018230 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.017959 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:20.018230 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018013 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-slash\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018230 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.018036 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs podName:73836491-7624-4e24-a8cd-2ad60bc09155 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:20.518009789 +0000 UTC m=+3.126984900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs") pod "network-metrics-daemon-zr6kd" (UID: "73836491-7624-4e24-a8cd-2ad60bc09155") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:20.018230 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018068 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-run-openvswitch\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018230 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018097 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-ovnkube-config\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018255 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b43f76a2-cd92-49a6-b567-71f3ad019d30-kubelet-config\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2hxx\" (UniqueName: \"kubernetes.io/projected/4702c99f-9747-4b1d-853d-090e6a7327ba-kube-api-access-g2hxx\") pod \"iptables-alerter-rhbv5\" (UID: \"4702c99f-9747-4b1d-853d-090e6a7327ba\") " pod="openshift-network-operator/iptables-alerter-rhbv5" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dea8be0d-b3bc-4bb9-80d5-00bf1d08a034-agent-certs\") pod \"konnectivity-agent-xpmql\" (UID: \"dea8be0d-b3bc-4bb9-80d5-00bf1d08a034\") " pod="kube-system/konnectivity-agent-xpmql" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4702c99f-9747-4b1d-853d-090e6a7327ba-iptables-alerter-script\") pod \"iptables-alerter-rhbv5\" (UID: \"4702c99f-9747-4b1d-853d-090e6a7327ba\") " pod="openshift-network-operator/iptables-alerter-rhbv5" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-run-ovn\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-log-socket\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-log-socket\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018559 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-run-ovn\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018602 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-ovnkube-script-lib\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018621 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b43f76a2-cd92-49a6-b567-71f3ad019d30-kubelet-config\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-cni-bin\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13432308-4a00-44e2-babf-6daaff232a0a-host\") pod \"node-ca-brf5d\" (UID: \"13432308-4a00-44e2-babf-6daaff232a0a\") " pod="openshift-image-registry/node-ca-brf5d" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018675 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-etc-openvswitch\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xt7z\" (UniqueName: \"kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z\") pod \"network-check-target-dvr7g\" (UID: \"dbfac494-fbe8-4979-ba88-fcf267911128\") " pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.018758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018743 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dea8be0d-b3bc-4bb9-80d5-00bf1d08a034-konnectivity-ca\") pod \"konnectivity-agent-xpmql\" (UID: \"dea8be0d-b3bc-4bb9-80d5-00bf1d08a034\") " pod="kube-system/konnectivity-agent-xpmql" Sep 05 07:48:20.019624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018745 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/13432308-4a00-44e2-babf-6daaff232a0a-host\") pod \"node-ca-brf5d\" (UID: \"13432308-4a00-44e2-babf-6daaff232a0a\") " pod="openshift-image-registry/node-ca-brf5d" Sep 05 07:48:20.019624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-env-overrides\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.019624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-cni-bin\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.019624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:20.019624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.018865 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhnz\" (UniqueName: \"kubernetes.io/projected/73836491-7624-4e24-a8cd-2ad60bc09155-kube-api-access-wwhnz\") pod \"network-metrics-daemon-zr6kd\" (UID: \"73836491-7624-4e24-a8cd-2ad60bc09155\") " pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:20.019624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.019038 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4702c99f-9747-4b1d-853d-090e6a7327ba-iptables-alerter-script\") pod \"iptables-alerter-rhbv5\" (UID: \"4702c99f-9747-4b1d-853d-090e6a7327ba\") " pod="openshift-network-operator/iptables-alerter-rhbv5" Sep 05 07:48:20.019624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.019068 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-env-overrides\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.019624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.019107 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-etc-openvswitch\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.019624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.019134 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.019624 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.019148 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:20.019624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.019158 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.019624 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.019214 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret podName:b43f76a2-cd92-49a6-b567-71f3ad019d30 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:20.519195325 +0000 UTC m=+3.128170439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret") pod "global-pull-secret-syncer-xtq7n" (UID: "b43f76a2-cd92-49a6-b567-71f3ad019d30") : object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:20.019624 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.019501 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/dea8be0d-b3bc-4bb9-80d5-00bf1d08a034-konnectivity-ca\") pod \"konnectivity-agent-xpmql\" (UID: \"dea8be0d-b3bc-4bb9-80d5-00bf1d08a034\") " pod="kube-system/konnectivity-agent-xpmql" Sep 05 07:48:20.022225 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.022201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-ovn-node-metrics-cert\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.023011 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.022932 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/dea8be0d-b3bc-4bb9-80d5-00bf1d08a034-agent-certs\") pod \"konnectivity-agent-xpmql\" (UID: \"dea8be0d-b3bc-4bb9-80d5-00bf1d08a034\") " pod="kube-system/konnectivity-agent-xpmql" Sep 05 07:48:20.028892 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.028141 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lz68\" (UniqueName: \"kubernetes.io/projected/13432308-4a00-44e2-babf-6daaff232a0a-kube-api-access-7lz68\") pod \"node-ca-brf5d\" (UID: \"13432308-4a00-44e2-babf-6daaff232a0a\") " pod="openshift-image-registry/node-ca-brf5d" Sep 05 07:48:20.032468 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.032377 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m4tw\" (UniqueName: \"kubernetes.io/projected/b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb-kube-api-access-7m4tw\") pod \"ovnkube-node-zbn86\" (UID: \"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.032703 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.032675 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svxtm\" (UniqueName: \"kubernetes.io/projected/a6bf702d-539a-4164-9e92-18a93205e30f-kube-api-access-svxtm\") pod \"node-resolver-4bmbd\" (UID: \"a6bf702d-539a-4164-9e92-18a93205e30f\") " pod="openshift-dns/node-resolver-4bmbd" Sep 05 07:48:20.032799 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.032756 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 07:48:20.032799 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.032777 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 07:48:20.032799 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.032791 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xt7z for pod openshift-network-diagnostics/network-check-target-dvr7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:20.032921 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.032857 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z podName:dbfac494-fbe8-4979-ba88-fcf267911128 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:20.532837899 +0000 UTC m=+3.141813017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5xt7z" (UniqueName: "kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z") pod "network-check-target-dvr7g" (UID: "dbfac494-fbe8-4979-ba88-fcf267911128") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:20.039826 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.039800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2hxx\" (UniqueName: \"kubernetes.io/projected/4702c99f-9747-4b1d-853d-090e6a7327ba-kube-api-access-g2hxx\") pod \"iptables-alerter-rhbv5\" (UID: \"4702c99f-9747-4b1d-853d-090e6a7327ba\") " pod="openshift-network-operator/iptables-alerter-rhbv5" Sep 05 07:48:20.040012 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.039991 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhnz\" (UniqueName: \"kubernetes.io/projected/73836491-7624-4e24-a8cd-2ad60bc09155-kube-api-access-wwhnz\") pod \"network-metrics-daemon-zr6kd\" (UID: \"73836491-7624-4e24-a8cd-2ad60bc09155\") " pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:20.112825 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.112794 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dsknf" Sep 05 07:48:20.125031 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.125006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l224c" Sep 05 07:48:20.142304 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.142279 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" Sep 05 07:48:20.150230 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.150193 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" Sep 05 07:48:20.158990 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.158964 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4bmbd" Sep 05 07:48:20.168028 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.168002 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-brf5d" Sep 05 07:48:20.178848 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.178826 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rhbv5" Sep 05 07:48:20.192771 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.192747 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:20.194926 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.194907 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xpmql" Sep 05 07:48:20.363228 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.363127 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 07:48:20.523137 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.523091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs\") pod \"network-metrics-daemon-zr6kd\" (UID: \"73836491-7624-4e24-a8cd-2ad60bc09155\") " pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:20.523367 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.523168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:20.523367 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.523313 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:20.523367 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.523331 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:20.523532 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.523383 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret podName:b43f76a2-cd92-49a6-b567-71f3ad019d30 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:21.523362721 +0000 UTC m=+4.132337825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret") pod "global-pull-secret-syncer-xtq7n" (UID: "b43f76a2-cd92-49a6-b567-71f3ad019d30") : object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:20.523532 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.523403 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs podName:73836491-7624-4e24-a8cd-2ad60bc09155 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:21.523394935 +0000 UTC m=+4.132370037 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs") pod "network-metrics-daemon-zr6kd" (UID: "73836491-7624-4e24-a8cd-2ad60bc09155") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:20.573306 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.573271 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 05 07:48:20.623792 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.623698 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xt7z\" (UniqueName: \"kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z\") pod \"network-check-target-dvr7g\" (UID: \"dbfac494-fbe8-4979-ba88-fcf267911128\") " pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:20.623985 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.623859 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 07:48:20.623985 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.623887 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 07:48:20.623985 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.623898 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xt7z for pod openshift-network-diagnostics/network-check-target-dvr7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:20.623985 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.623955 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z podName:dbfac494-fbe8-4979-ba88-fcf267911128 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:21.623938229 +0000 UTC m=+4.232913331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xt7z" (UniqueName: "kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z") pod "network-check-target-dvr7g" (UID: "dbfac494-fbe8-4979-ba88-fcf267911128") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:20.880134 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.880046 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:20.880698 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.880065 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:20.880698 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.880172 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:20.880698 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:20.880300 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:20.931633 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.931587 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 07:43:20 +0000 UTC" deadline="2028-02-06 17:31:59.757849196 +0000 UTC" Sep 05 07:48:20.931633 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:20.931625 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12465h43m38.826228314s" Sep 05 07:48:21.144697 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:21.144659 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d05ed4_423c_4619_89c1_3d4a00b8ea86.slice/crio-a16262e7492c0e67e99bf5262f14bd3b3fb3e93eb70589e4092f75916887cc4a WatchSource:0}: Error finding container a16262e7492c0e67e99bf5262f14bd3b3fb3e93eb70589e4092f75916887cc4a: Status 404 returned error can't find the container with id a16262e7492c0e67e99bf5262f14bd3b3fb3e93eb70589e4092f75916887cc4a Sep 05 07:48:21.182644 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:21.182614 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4702c99f_9747_4b1d_853d_090e6a7327ba.slice/crio-c74eb3308b31471567e641a8328b8cacaf4a346e1358fd32198c7f65c16ffb7a WatchSource:0}: Error finding container c74eb3308b31471567e641a8328b8cacaf4a346e1358fd32198c7f65c16ffb7a: Status 404 returned error can't find the container with id c74eb3308b31471567e641a8328b8cacaf4a346e1358fd32198c7f65c16ffb7a Sep 05 07:48:21.183732 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:21.183705 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dc39b8_d4d0_469d_8d91_f7d4ed7265cb.slice/crio-376eb3cdbb8449dfe585341f918eb10b2e9a37bf12af89fc0e66ea3a07e2fa72 WatchSource:0}: Error finding container 376eb3cdbb8449dfe585341f918eb10b2e9a37bf12af89fc0e66ea3a07e2fa72: Status 404 returned error can't find the container with id 376eb3cdbb8449dfe585341f918eb10b2e9a37bf12af89fc0e66ea3a07e2fa72 Sep 05 07:48:21.184885 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:21.184847 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bd7c66_b8fe_40ea_869c_4a8dd1c743dd.slice/crio-3f632f91a592da74f844e8a80a1c9f2782f4b7e28bf5b98557c5e1a6a251af5b WatchSource:0}: Error finding container 3f632f91a592da74f844e8a80a1c9f2782f4b7e28bf5b98557c5e1a6a251af5b: Status 404 returned error can't find the container with id 3f632f91a592da74f844e8a80a1c9f2782f4b7e28bf5b98557c5e1a6a251af5b Sep 05 07:48:21.185917 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:21.185893 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod861a7bc4_b5ce_4432_9ba4_c2f938aab3f3.slice/crio-3f7789fae46f187b0ac6e0be8d80e6c0c9bddcf32033dd4a4d27acff0c234cac WatchSource:0}: Error finding container 3f7789fae46f187b0ac6e0be8d80e6c0c9bddcf32033dd4a4d27acff0c234cac: Status 404 returned error can't find the container with id 3f7789fae46f187b0ac6e0be8d80e6c0c9bddcf32033dd4a4d27acff0c234cac Sep 05 07:48:21.531770 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.531738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:21.531912 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.531800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs\") pod \"network-metrics-daemon-zr6kd\" (UID: \"73836491-7624-4e24-a8cd-2ad60bc09155\") " pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:21.531970 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:21.531918 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:21.531970 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:21.531918 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:21.532063 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:21.531994 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs podName:73836491-7624-4e24-a8cd-2ad60bc09155 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:23.531973069 +0000 UTC m=+6.140948173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs") pod "network-metrics-daemon-zr6kd" (UID: "73836491-7624-4e24-a8cd-2ad60bc09155") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:21.532063 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:21.532015 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret podName:b43f76a2-cd92-49a6-b567-71f3ad019d30 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:23.532005848 +0000 UTC m=+6.140980951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret") pod "global-pull-secret-syncer-xtq7n" (UID: "b43f76a2-cd92-49a6-b567-71f3ad019d30") : object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:21.632962 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.632925 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xt7z\" (UniqueName: \"kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z\") pod \"network-check-target-dvr7g\" (UID: \"dbfac494-fbe8-4979-ba88-fcf267911128\") " pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:21.633133 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:21.633114 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 07:48:21.633173 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:21.633140 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 07:48:21.633173 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:21.633150 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xt7z for pod openshift-network-diagnostics/network-check-target-dvr7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:21.633248 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:21.633202 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z podName:dbfac494-fbe8-4979-ba88-fcf267911128 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:23.633187935 +0000 UTC m=+6.242163042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xt7z" (UniqueName: "kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z") pod "network-check-target-dvr7g" (UID: "dbfac494-fbe8-4979-ba88-fcf267911128") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:21.883127 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.883046 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:21.884111 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:21.883221 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:21.911214 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.911169 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xpmql" event={"ID":"dea8be0d-b3bc-4bb9-80d5-00bf1d08a034","Type":"ContainerStarted","Data":"4e053d16b09459025c0f309ff5f05127a7bbc7ec00e63b0218fc38759bc012c0"} Sep 05 07:48:21.925229 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.925188 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" event={"ID":"f7fd8127-7a31-414b-962a-84b1e0303114","Type":"ContainerStarted","Data":"e1b77f7688805ccb19a48d843862fcaef3d226dcc08195f6617281d92914eb00"} Sep 05 07:48:21.933065 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.933024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsknf" event={"ID":"e1d05ed4-423c-4619-89c1-3d4a00b8ea86","Type":"ContainerStarted","Data":"a16262e7492c0e67e99bf5262f14bd3b3fb3e93eb70589e4092f75916887cc4a"} Sep 05 07:48:21.933330 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.933205 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-04 07:43:20 +0000 UTC" deadline="2028-05-01 01:42:43.19633821 +0000 UTC" Sep 05 07:48:21.933330 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.933244 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14489h54m21.263097869s" Sep 05 07:48:21.939312 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.938140 2576 generic.go:356] "Generic (PLEG): container finished" podID="00a91b5c1abc892797819c828b866aed" containerID="ed25860e65a1841f8446ddb4f4c1ea1c17d42fad0c01e65ce33b411b86a80839" exitCode=0 Sep 05 07:48:21.940307 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.939460 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal" event={"ID":"00a91b5c1abc892797819c828b866aed","Type":"ContainerDied","Data":"ed25860e65a1841f8446ddb4f4c1ea1c17d42fad0c01e65ce33b411b86a80839"} Sep 05 07:48:21.945167 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.945137 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" event={"ID":"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb","Type":"ContainerStarted","Data":"376eb3cdbb8449dfe585341f918eb10b2e9a37bf12af89fc0e66ea3a07e2fa72"} Sep 05 07:48:21.958414 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.958376 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rhbv5" event={"ID":"4702c99f-9747-4b1d-853d-090e6a7327ba","Type":"ContainerStarted","Data":"c74eb3308b31471567e641a8328b8cacaf4a346e1358fd32198c7f65c16ffb7a"} Sep 05 07:48:21.963437 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.963408 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4bmbd" event={"ID":"a6bf702d-539a-4164-9e92-18a93205e30f","Type":"ContainerStarted","Data":"3b6e77eae41a2a349c8c98fee0bb16ac5d10e06d87e75141f64b333523a4ac7b"} Sep 05 07:48:21.968523 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.968492 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-brf5d" event={"ID":"13432308-4a00-44e2-babf-6daaff232a0a","Type":"ContainerStarted","Data":"46ce1110448b07a64aa900c33079c244926b0aea5ca89bf974fc83d13cb19fac"} Sep 05 07:48:21.984490 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:21.984448 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-184.ec2.internal" event={"ID":"76bb8a07519e0ff54e97f8d73d9e9f48","Type":"ContainerStarted","Data":"0a1e33d86ad5ef1d84ca4812d50799469b88bb98e3028e3fc0e3aebc1c21efd4"} Sep 05 07:48:22.002239 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:22.002018 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-184.ec2.internal" podStartSLOduration=2.999077517 podStartE2EDuration="2.999077517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 07:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:48:21.998583112 +0000 UTC m=+4.607558241" watchObservedRunningTime="2026-09-05 07:48:21.999077517 +0000 UTC m=+4.608052644" Sep 05 07:48:22.013175 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:22.013085 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l224c" event={"ID":"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3","Type":"ContainerStarted","Data":"3f7789fae46f187b0ac6e0be8d80e6c0c9bddcf32033dd4a4d27acff0c234cac"} Sep 05 07:48:22.026463 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:22.026370 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" event={"ID":"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd","Type":"ContainerStarted","Data":"3f632f91a592da74f844e8a80a1c9f2782f4b7e28bf5b98557c5e1a6a251af5b"} Sep 05 07:48:22.880779 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:22.880749 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:22.880959 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:22.880882 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:22.881036 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:22.880964 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:22.881094 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:22.881036 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:23.047407 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:23.047374 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal" event={"ID":"00a91b5c1abc892797819c828b866aed","Type":"ContainerStarted","Data":"a24d27d4327fab70aa38b2df25b25f0ff57d1f6cb5aa7d4f8acb26ceba36c654"} Sep 05 07:48:23.063832 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:23.063776 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-184.ec2.internal" podStartSLOduration=4.06375322 podStartE2EDuration="4.06375322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 07:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:48:23.062538397 +0000 UTC m=+5.671513523" watchObservedRunningTime="2026-09-05 07:48:23.06375322 +0000 UTC m=+5.672728344" Sep 05 07:48:23.551319 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:23.550968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs\") pod \"network-metrics-daemon-zr6kd\" (UID: \"73836491-7624-4e24-a8cd-2ad60bc09155\") " pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:23.551319 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:23.551049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:23.551319 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:23.551124 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:23.551319 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:23.551172 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:23.551319 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:23.551187 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs podName:73836491-7624-4e24-a8cd-2ad60bc09155 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:27.551170773 +0000 UTC m=+10.160145877 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs") pod "network-metrics-daemon-zr6kd" (UID: "73836491-7624-4e24-a8cd-2ad60bc09155") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:23.551319 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:23.551245 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret podName:b43f76a2-cd92-49a6-b567-71f3ad019d30 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:27.551226509 +0000 UTC m=+10.160201624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret") pod "global-pull-secret-syncer-xtq7n" (UID: "b43f76a2-cd92-49a6-b567-71f3ad019d30") : object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:23.651752 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:23.651714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xt7z\" (UniqueName: \"kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z\") pod \"network-check-target-dvr7g\" (UID: \"dbfac494-fbe8-4979-ba88-fcf267911128\") " pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:23.651940 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:23.651888 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 07:48:23.651940 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:23.651909 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 07:48:23.651940 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:23.651921 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xt7z for pod openshift-network-diagnostics/network-check-target-dvr7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:23.652085 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:23.652007 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z podName:dbfac494-fbe8-4979-ba88-fcf267911128 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:27.65198699 +0000 UTC m=+10.260962107 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xt7z" (UniqueName: "kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z") pod "network-check-target-dvr7g" (UID: "dbfac494-fbe8-4979-ba88-fcf267911128") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:23.880507 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:23.880422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:23.880674 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:23.880576 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:24.880181 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:24.880144 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:24.880625 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:24.880300 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:24.880625 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:24.880600 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:24.880692 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:24.880668 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:25.882650 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:25.881383 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:25.882650 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:25.881534 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:26.881640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:26.881071 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:26.881640 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:26.881199 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:26.881640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:26.881269 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:26.881640 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:26.881380 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:27.587642 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:27.587609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs\") pod \"network-metrics-daemon-zr6kd\" (UID: \"73836491-7624-4e24-a8cd-2ad60bc09155\") " pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:27.588084 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:27.587661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:27.588084 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:27.587750 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:27.588084 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:27.587794 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret podName:b43f76a2-cd92-49a6-b567-71f3ad019d30 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:35.587781134 +0000 UTC m=+18.196756236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret") pod "global-pull-secret-syncer-xtq7n" (UID: "b43f76a2-cd92-49a6-b567-71f3ad019d30") : object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:27.588243 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:27.588106 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:27.588243 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:27.588141 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs podName:73836491-7624-4e24-a8cd-2ad60bc09155 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:35.588130315 +0000 UTC m=+18.197105415 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs") pod "network-metrics-daemon-zr6kd" (UID: "73836491-7624-4e24-a8cd-2ad60bc09155") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:27.687965 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:27.687932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xt7z\" (UniqueName: \"kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z\") pod \"network-check-target-dvr7g\" (UID: \"dbfac494-fbe8-4979-ba88-fcf267911128\") " pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:27.688253 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:27.688232 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 07:48:27.688342 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:27.688260 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 07:48:27.688342 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:27.688272 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xt7z for pod openshift-network-diagnostics/network-check-target-dvr7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:27.688342 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:27.688328 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z podName:dbfac494-fbe8-4979-ba88-fcf267911128 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:35.688309226 +0000 UTC m=+18.297284338 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xt7z" (UniqueName: "kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z") pod "network-check-target-dvr7g" (UID: "dbfac494-fbe8-4979-ba88-fcf267911128") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:27.886192 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:27.886117 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:27.886370 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:27.886266 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:28.880364 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:28.880329 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:28.880815 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:28.880330 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:28.880815 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:28.880462 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:28.880815 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:28.880534 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:29.883123 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:29.883040 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:29.883581 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:29.883179 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:30.880898 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:30.880850 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:30.880898 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:30.880866 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:30.881164 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:30.880964 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:30.881164 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:30.881111 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:31.880700 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:31.880613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:31.881132 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:31.880752 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:32.880937 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:32.880901 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:32.881417 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:32.880905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:32.881417 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:32.881015 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:32.881417 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:32.881096 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:33.880421 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:33.880388 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:33.880588 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:33.880537 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:34.880230 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:34.880177 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:34.880230 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:34.880221 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:34.880696 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:34.880330 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:34.880696 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:34.880498 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:35.650440 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:35.650351 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs\") pod \"network-metrics-daemon-zr6kd\" (UID: \"73836491-7624-4e24-a8cd-2ad60bc09155\") " pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:35.650440 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:35.650422 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:35.650649 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:35.650526 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:35.650649 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:35.650534 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:35.650649 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:35.650604 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret podName:b43f76a2-cd92-49a6-b567-71f3ad019d30 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:51.650584723 +0000 UTC m=+34.259559842 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret") pod "global-pull-secret-syncer-xtq7n" (UID: "b43f76a2-cd92-49a6-b567-71f3ad019d30") : object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:35.650649 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:35.650624 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs podName:73836491-7624-4e24-a8cd-2ad60bc09155 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:51.650614351 +0000 UTC m=+34.259589473 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs") pod "network-metrics-daemon-zr6kd" (UID: "73836491-7624-4e24-a8cd-2ad60bc09155") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:35.751321 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:35.751278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xt7z\" (UniqueName: \"kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z\") pod \"network-check-target-dvr7g\" (UID: \"dbfac494-fbe8-4979-ba88-fcf267911128\") " pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:35.751509 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:35.751491 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 07:48:35.751509 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:35.751510 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 07:48:35.751615 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:35.751519 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xt7z for pod openshift-network-diagnostics/network-check-target-dvr7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:35.751615 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:35.751573 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z podName:dbfac494-fbe8-4979-ba88-fcf267911128 nodeName:}" failed. No retries permitted until 2026-09-05 07:48:51.751560001 +0000 UTC m=+34.360535124 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xt7z" (UniqueName: "kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z") pod "network-check-target-dvr7g" (UID: "dbfac494-fbe8-4979-ba88-fcf267911128") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:35.880891 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:35.880862 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:35.881360 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:35.881003 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:36.880965 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:36.880924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:36.881384 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:36.881062 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:36.881384 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:36.881107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:36.881384 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:36.881190 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:37.881204 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:37.881172 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:37.881645 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:37.881318 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:38.880868 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:38.880833 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:38.881027 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:38.880837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:38.881027 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:38.880942 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:38.881138 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:38.881047 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:39.880788 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:39.880620 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:39.881656 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:39.880871 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:40.085403 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.085107 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xpmql" event={"ID":"dea8be0d-b3bc-4bb9-80d5-00bf1d08a034","Type":"ContainerStarted","Data":"68a36775c18e982e1b9c4a09f76ce7fdcd7a8eabb7ac3e259caf5eab239c145a"} Sep 05 07:48:40.087272 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.087232 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" event={"ID":"f7fd8127-7a31-414b-962a-84b1e0303114","Type":"ContainerStarted","Data":"20cbb219d6fe3013376e54141e29ac00a2fded48cc632395233ee9c0fe810203"} Sep 05 07:48:40.089283 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.089254 2576 generic.go:356] "Generic (PLEG): container finished" podID="e1d05ed4-423c-4619-89c1-3d4a00b8ea86" containerID="437677d4293f3c2347428c7d1601131c7ec0f6c391a679457d72ca24163b6ade" exitCode=0 Sep 05 07:48:40.089418 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.089349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsknf" event={"ID":"e1d05ed4-423c-4619-89c1-3d4a00b8ea86","Type":"ContainerDied","Data":"437677d4293f3c2347428c7d1601131c7ec0f6c391a679457d72ca24163b6ade"} Sep 05 07:48:40.097988 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.097968 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 07:48:40.098465 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.098437 2576 generic.go:356] "Generic (PLEG): container finished" podID="b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb" containerID="ab794365c43d900deeab36b628b3dfa700da60f0319e5d9be3111a71c1a29c72" exitCode=1 Sep 05 07:48:40.098548 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.098507 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" event={"ID":"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb","Type":"ContainerStarted","Data":"fe79c58d8a7a979d568d4367a7598ebe4bf25b539ced6afe0141e41867ec3de9"} Sep 05 07:48:40.098630 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.098612 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" event={"ID":"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb","Type":"ContainerStarted","Data":"8d60ceaf33837b98dc3d587695d8f963ba8af99d17ee1e3d2f27357f60e9c26c"} Sep 05 07:48:40.098733 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.098667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" event={"ID":"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb","Type":"ContainerStarted","Data":"f2438c40bb731082088d42f094e7dd4e38ca71ee046d6ce448b7976bee398ce2"} Sep 05 07:48:40.098733 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.098715 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" event={"ID":"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb","Type":"ContainerStarted","Data":"7e46650335e92106ef08b30f4c4f2375ba55f1b838a815627cd305c67042c972"} Sep 05 07:48:40.098824 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.098763 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" event={"ID":"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb","Type":"ContainerDied","Data":"ab794365c43d900deeab36b628b3dfa700da60f0319e5d9be3111a71c1a29c72"} Sep 05 07:48:40.098824 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.098811 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" event={"ID":"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb","Type":"ContainerStarted","Data":"ed69462ad27a454fd23686da8cf18da40aae9283982dad54233ec03ef1cd3976"} Sep 05 07:48:40.100338 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.100278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4bmbd" event={"ID":"a6bf702d-539a-4164-9e92-18a93205e30f","Type":"ContainerStarted","Data":"87413d0a77ef393644b41c25d0b54e1916f292d111480196ffa24197afd804b7"} Sep 05 07:48:40.101962 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.101941 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-brf5d" event={"ID":"13432308-4a00-44e2-babf-6daaff232a0a","Type":"ContainerStarted","Data":"afd971af7c342d673fdfec4add8f7029ba3748384fd2da54c39145f35e2b7d2d"} Sep 05 07:48:40.103749 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.103692 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l224c" event={"ID":"861a7bc4-b5ce-4432-9ba4-c2f938aab3f3","Type":"ContainerStarted","Data":"49330c4f820cceae1c32bc6096493a0738714f45b77dc19cd8c62b0d9a1efa7a"} Sep 05 07:48:40.105504 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.105483 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" event={"ID":"b2bd7c66-b8fe-40ea-869c-4a8dd1c743dd","Type":"ContainerStarted","Data":"46d5b0333856d621de28388886fa3453654de1335dd05712233469f40aaf9d4e"} Sep 05 07:48:40.125447 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.125410 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xpmql" podStartSLOduration=8.038788179 podStartE2EDuration="22.125398689s" totalImagesPullingTime="14.08661051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:48:40.103594631 +0000 UTC m=+22.712569756" watchObservedRunningTime="2026-09-05 07:48:40.125398689 +0000 UTC m=+22.734373812" Sep 05 07:48:40.142949 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.142889 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4bmbd" podStartSLOduration=4.099678114 podStartE2EDuration="22.142869296s" totalImagesPullingTime="18.043191182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 07:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:48:40.125311248 +0000 UTC m=+22.734286372" watchObservedRunningTime="2026-09-05 07:48:40.142869296 +0000 UTC m=+22.751844420" Sep 05 07:48:40.143062 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.143003 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-brf5d" podStartSLOduration=4.072778648 podStartE2EDuration="22.142995217s" totalImagesPullingTime="18.070216569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 07:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:48:40.142112989 +0000 UTC m=+22.751088140" watchObservedRunningTime="2026-09-05 07:48:40.142995217 +0000 UTC m=+22.751970341" Sep 05 07:48:40.186588 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.186538 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vhlrm" podStartSLOduration=4.150859173 podStartE2EDuration="22.186524312s" totalImagesPullingTime="18.035665139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 07:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:48:40.185844787 +0000 UTC m=+22.794819912" watchObservedRunningTime="2026-09-05 07:48:40.186524312 +0000 UTC m=+22.795499435" Sep 05 07:48:40.204633 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.204581 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l224c" podStartSLOduration=3.93283209 podStartE2EDuration="22.20456774s" totalImagesPullingTime="18.27173565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 07:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:48:40.203970238 +0000 UTC m=+22.812945362" watchObservedRunningTime="2026-09-05 07:48:40.20456774 +0000 UTC m=+22.813542859" Sep 05 07:48:40.826521 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.826494 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 05 07:48:40.880088 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.880059 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:40.880285 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.880065 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:40.880285 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:40.880168 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:40.880388 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:40.880284 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:40.882126 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.882061 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-05T07:48:40.826517418Z","UUID":"be20cdb3-90bb-4cfd-ad13-a3a9b7a495b2","Handler":null,"Name":"","Endpoint":""} Sep 05 07:48:40.883657 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.883640 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 05 07:48:40.883743 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:40.883670 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 05 07:48:41.109417 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:41.109379 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rhbv5" event={"ID":"4702c99f-9747-4b1d-853d-090e6a7327ba","Type":"ContainerStarted","Data":"c3aca676ec9a6deb50a534435c990a103d627490b2949989e3cfccb2047b244c"} Sep 05 07:48:41.111508 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:41.111479 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" event={"ID":"f7fd8127-7a31-414b-962a-84b1e0303114","Type":"ContainerStarted","Data":"97269a234cec489125f75426ca702d0bdce155d3b4d66b6c3377a9facc458c7c"} Sep 05 07:48:41.123312 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:41.123266 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rhbv5" podStartSLOduration=5.084878611 podStartE2EDuration="23.123247643s" totalImagesPullingTime="18.038369032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 07:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:48:41.122908877 +0000 UTC m=+23.731884011" watchObservedRunningTime="2026-09-05 07:48:41.123247643 +0000 UTC m=+23.732222768" Sep 05 07:48:41.880687 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:41.880647 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:41.880836 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:41.880800 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:42.117665 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:42.117622 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" event={"ID":"f7fd8127-7a31-414b-962a-84b1e0303114","Type":"ContainerStarted","Data":"530d20524db40eeab8c8c8323464ffd732e44f87e91b8d8aff1ef9200028d6b9"} Sep 05 07:48:42.121718 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:42.121696 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 07:48:42.122285 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:42.122254 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" event={"ID":"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb","Type":"ContainerStarted","Data":"a81cd3411f0d229016eb0a0615c15e854331dd271ccf96a680051372e265b513"} Sep 05 07:48:42.134791 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:42.134735 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vck7d" podStartSLOduration=3.9745609330000002 podStartE2EDuration="24.134715901s" totalImagesPullingTime="20.160154968s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 07:48:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:48:42.133923968 +0000 UTC m=+24.742899093" watchObservedRunningTime="2026-09-05 07:48:42.134715901 +0000 UTC m=+24.743691024" Sep 05 07:48:42.880742 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:42.880716 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:42.880742 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:42.880751 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:42.880934 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:42.880837 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:42.880978 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:42.880939 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:43.127977 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:43.127939 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xpmql" Sep 05 07:48:43.128534 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:43.128516 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xpmql" Sep 05 07:48:43.880568 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:43.880536 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:43.880738 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:43.880675 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:44.148712 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:44.148633 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xpmql" Sep 05 07:48:44.149346 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:44.149323 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xpmql" Sep 05 07:48:44.880728 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:44.880531 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:44.880860 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:44.880612 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:44.880860 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:44.880827 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:44.880950 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:44.880871 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:45.132960 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:45.132886 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 07:48:45.133369 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:45.133339 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" event={"ID":"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb","Type":"ContainerStarted","Data":"1fb0f26a847337c0380651741da0414ee093b00a6a65a086f72fc4804bf55228"} Sep 05 07:48:45.134038 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:45.133584 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:45.134038 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:45.133716 2576 scope.go:117] "RemoveContainer" containerID="ab794365c43d900deeab36b628b3dfa700da60f0319e5d9be3111a71c1a29c72" Sep 05 07:48:45.135498 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:45.135477 2576 generic.go:356] "Generic (PLEG): container finished" podID="e1d05ed4-423c-4619-89c1-3d4a00b8ea86" containerID="ab6c2804fc0eee52cb788090162a6ce3c7bd334223764543d6583f25dbef021a" exitCode=0 Sep 05 07:48:45.135587 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:45.135561 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsknf" event={"ID":"e1d05ed4-423c-4619-89c1-3d4a00b8ea86","Type":"ContainerDied","Data":"ab6c2804fc0eee52cb788090162a6ce3c7bd334223764543d6583f25dbef021a"} Sep 05 07:48:45.151033 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:45.151012 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:45.880482 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:45.880458 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:45.880641 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:45.880619 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:46.141325 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.141011 2576 generic.go:356] "Generic (PLEG): container finished" podID="e1d05ed4-423c-4619-89c1-3d4a00b8ea86" containerID="fb84c48df5a0a6f36e78d57112bac919538334d828dba2f57c4696547986db74" exitCode=0 Sep 05 07:48:46.141488 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.141091 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsknf" event={"ID":"e1d05ed4-423c-4619-89c1-3d4a00b8ea86","Type":"ContainerDied","Data":"fb84c48df5a0a6f36e78d57112bac919538334d828dba2f57c4696547986db74"} Sep 05 07:48:46.148049 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.148031 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 07:48:46.148483 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.148457 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" event={"ID":"b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb","Type":"ContainerStarted","Data":"0da35a204edb3048e579f91f35239a7c4b1969d8907bd4e8f8f7a2336231e113"} Sep 05 07:48:46.148577 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.148500 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 05 07:48:46.148712 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.148695 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:46.164480 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.164462 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:46.192319 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.192277 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" podStartSLOduration=10.117954424 podStartE2EDuration="28.192263276s" totalImagesPullingTime="18.074308852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 07:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:48:46.191704267 +0000 UTC m=+28.800679402" watchObservedRunningTime="2026-09-05 07:48:46.192263276 +0000 UTC m=+28.801238399" Sep 05 07:48:46.329648 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.329616 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xtq7n"] Sep 05 07:48:46.329812 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.329763 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:46.329933 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:46.329909 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:46.332505 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.332471 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dvr7g"] Sep 05 07:48:46.332689 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.332568 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:46.332689 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:46.332666 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:46.342744 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.342719 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zr6kd"] Sep 05 07:48:46.342866 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:46.342824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:46.342940 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:46.342922 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:47.153014 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:47.152984 2576 generic.go:356] "Generic (PLEG): container finished" podID="e1d05ed4-423c-4619-89c1-3d4a00b8ea86" containerID="f147bfc121646a1c06849e6dcbcc76d6209020a9a25e33aa8d040c96fe831d53" exitCode=0 Sep 05 07:48:47.153154 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:47.153066 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsknf" event={"ID":"e1d05ed4-423c-4619-89c1-3d4a00b8ea86","Type":"ContainerDied","Data":"f147bfc121646a1c06849e6dcbcc76d6209020a9a25e33aa8d040c96fe831d53"} Sep 05 07:48:47.153154 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:47.153137 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 05 07:48:47.673393 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:47.673357 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:48:47.881175 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:47.881149 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:47.881386 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:47.881250 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:47.881386 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:47.881274 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:47.881386 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:47.881267 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:47.881386 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:47.881355 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:47.881576 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:47.881446 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:49.169042 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:49.168992 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" podUID="b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb" containerName="ovnkube-controller" probeResult="failure" output="" Sep 05 07:48:49.880559 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:49.880530 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:49.880559 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:49.880547 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:49.880766 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:49.880529 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:49.880766 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:49.880665 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:49.880910 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:49.880783 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:49.880910 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:49.880880 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:51.672229 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:51.672018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs\") pod \"network-metrics-daemon-zr6kd\" (UID: \"73836491-7624-4e24-a8cd-2ad60bc09155\") " pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:51.672606 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:51.672280 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:51.672606 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:51.672162 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:51.672606 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:51.672349 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs podName:73836491-7624-4e24-a8cd-2ad60bc09155 nodeName:}" failed. No retries permitted until 2026-09-05 07:49:23.672332285 +0000 UTC m=+66.281307391 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs") pod "network-metrics-daemon-zr6kd" (UID: "73836491-7624-4e24-a8cd-2ad60bc09155") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 05 07:48:51.672606 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:51.672382 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:51.672606 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:51.672429 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret podName:b43f76a2-cd92-49a6-b567-71f3ad019d30 nodeName:}" failed. No retries permitted until 2026-09-05 07:49:23.672416354 +0000 UTC m=+66.281391456 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret") pod "global-pull-secret-syncer-xtq7n" (UID: "b43f76a2-cd92-49a6-b567-71f3ad019d30") : object "kube-system"/"original-pull-secret" not registered Sep 05 07:48:51.773285 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:51.773198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xt7z\" (UniqueName: \"kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z\") pod \"network-check-target-dvr7g\" (UID: \"dbfac494-fbe8-4979-ba88-fcf267911128\") " pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:51.773400 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:51.773381 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 05 07:48:51.773436 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:51.773405 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 05 07:48:51.773436 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:51.773418 2576 projected.go:196] Error preparing data for projected volume kube-api-access-5xt7z for pod openshift-network-diagnostics/network-check-target-dvr7g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:51.773497 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:51.773469 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z podName:dbfac494-fbe8-4979-ba88-fcf267911128 nodeName:}" failed. No retries permitted until 2026-09-05 07:49:23.773455841 +0000 UTC m=+66.382430961 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5xt7z" (UniqueName: "kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z") pod "network-check-target-dvr7g" (UID: "dbfac494-fbe8-4979-ba88-fcf267911128") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 05 07:48:51.880715 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:51.880693 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:51.880821 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:51.880699 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:51.880856 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:51.880801 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xtq7n" podUID="b43f76a2-cd92-49a6-b567-71f3ad019d30" Sep 05 07:48:51.880856 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:51.880842 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:51.880943 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:51.880927 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zr6kd" podUID="73836491-7624-4e24-a8cd-2ad60bc09155" Sep 05 07:48:51.881005 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:51.880988 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvr7g" podUID="dbfac494-fbe8-4979-ba88-fcf267911128" Sep 05 07:48:52.166961 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.166930 2576 generic.go:356] "Generic (PLEG): container finished" podID="e1d05ed4-423c-4619-89c1-3d4a00b8ea86" containerID="f9f4bca10ea779f95d225b5fd5f52ff40d6737468e44191fea3644edd92396cf" exitCode=0 Sep 05 07:48:52.166961 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.166967 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsknf" event={"ID":"e1d05ed4-423c-4619-89c1-3d4a00b8ea86","Type":"ContainerDied","Data":"f9f4bca10ea779f95d225b5fd5f52ff40d6737468e44191fea3644edd92396cf"} Sep 05 07:48:52.203761 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.203742 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-184.ec2.internal" event="NodeReady" Sep 05 07:48:52.203864 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.203859 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 05 07:48:52.236499 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.236472 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-859f44b44c-q9ptx"] Sep 05 07:48:52.271845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.271824 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-859f44b44c-q9ptx"] Sep 05 07:48:52.271845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.271855 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7cs9n"] Sep 05 07:48:52.271998 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.271972 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.274152 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.274130 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 05 07:48:52.274328 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.274179 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-z2wx8\"" Sep 05 07:48:52.274328 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.274229 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 05 07:48:52.274328 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.274260 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 05 07:48:52.279326 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.279305 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 05 07:48:52.287964 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.287943 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rbg4k"] Sep 05 07:48:52.288070 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.288058 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:52.289982 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.289957 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 05 07:48:52.289982 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.289979 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6wxv\"" Sep 05 07:48:52.290127 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.290066 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 05 07:48:52.314085 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.314059 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rbg4k"] Sep 05 07:48:52.314085 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.314090 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7cs9n"] Sep 05 07:48:52.314234 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.314175 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rbg4k" Sep 05 07:48:52.316170 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.316149 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 05 07:48:52.316170 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.316167 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 05 07:48:52.316344 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.316247 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 05 07:48:52.316344 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.316255 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qqkv2\"" Sep 05 07:48:52.376792 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.376768 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332ec72e-6421-48a2-b0c0-def5d946f80c-tmp-dir\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:52.376880 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.376803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-certificates\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.376880 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.376825 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9lcs\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-kube-api-access-q9lcs\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.376880 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.376841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-installation-pull-secrets\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.377022 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.376921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/332ec72e-6421-48a2-b0c0-def5d946f80c-config-volume\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:52.377022 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.376947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc78n\" (UniqueName: \"kubernetes.io/projected/332ec72e-6421-48a2-b0c0-def5d946f80c-kube-api-access-nc78n\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:52.377022 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.376967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:52.377118 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.377018 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.377118 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.377051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-trusted-ca\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.377118 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.377071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-bound-sa-token\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.377118 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.377100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-image-registry-private-configuration\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.377260 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.377118 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-ca-trust-extracted\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.478163 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-installation-pull-secrets\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.478163 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478138 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/332ec72e-6421-48a2-b0c0-def5d946f80c-config-volume\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:52.478163 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478157 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc78n\" (UniqueName: \"kubernetes.io/projected/332ec72e-6421-48a2-b0c0-def5d946f80c-kube-api-access-nc78n\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:52.478383 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478297 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:52.478383 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359-cert\") pod \"ingress-canary-rbg4k\" (UID: \"efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359\") " pod="openshift-ingress-canary/ingress-canary-rbg4k" Sep 05 07:48:52.478479 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.478479 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:52.478422 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 07:48:52.478564 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-trusted-ca\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.478564 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:52.478485 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls podName:332ec72e-6421-48a2-b0c0-def5d946f80c nodeName:}" failed. No retries permitted until 2026-09-05 07:48:52.97846424 +0000 UTC m=+35.587439347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls") pod "dns-default-7cs9n" (UID: "332ec72e-6421-48a2-b0c0-def5d946f80c") : secret "dns-default-metrics-tls" not found Sep 05 07:48:52.478564 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:52.478512 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 07:48:52.478564 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:52.478524 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859f44b44c-q9ptx: secret "image-registry-tls" not found Sep 05 07:48:52.478564 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:52.478560 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls podName:a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc nodeName:}" failed. No retries permitted until 2026-09-05 07:48:52.978547648 +0000 UTC m=+35.587522752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls") pod "image-registry-859f44b44c-q9ptx" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc") : secret "image-registry-tls" not found Sep 05 07:48:52.478564 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-bound-sa-token\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.478782 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478607 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-image-registry-private-configuration\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.478782 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-ca-trust-extracted\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.478782 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332ec72e-6421-48a2-b0c0-def5d946f80c-tmp-dir\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:52.478782 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-certificates\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.478782 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478706 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/332ec72e-6421-48a2-b0c0-def5d946f80c-config-volume\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:52.478782 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdrgh\" (UniqueName: \"kubernetes.io/projected/efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359-kube-api-access-wdrgh\") pod \"ingress-canary-rbg4k\" (UID: \"efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359\") " pod="openshift-ingress-canary/ingress-canary-rbg4k" Sep 05 07:48:52.478782 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9lcs\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-kube-api-access-q9lcs\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.479133 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.478954 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/332ec72e-6421-48a2-b0c0-def5d946f80c-tmp-dir\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:52.479179 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.479135 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-certificates\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.479179 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.479133 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-ca-trust-extracted\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.479600 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.479560 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-trusted-ca\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.482363 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.482347 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-image-registry-private-configuration\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.482416 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.482392 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-installation-pull-secrets\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.485989 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.485962 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc78n\" (UniqueName: \"kubernetes.io/projected/332ec72e-6421-48a2-b0c0-def5d946f80c-kube-api-access-nc78n\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:52.486275 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.486259 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-bound-sa-token\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.486515 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.486493 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9lcs\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-kube-api-access-q9lcs\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.579518 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.579487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359-cert\") pod \"ingress-canary-rbg4k\" (UID: \"efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359\") " pod="openshift-ingress-canary/ingress-canary-rbg4k" Sep 05 07:48:52.579669 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.579555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdrgh\" (UniqueName: \"kubernetes.io/projected/efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359-kube-api-access-wdrgh\") pod \"ingress-canary-rbg4k\" (UID: \"efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359\") " pod="openshift-ingress-canary/ingress-canary-rbg4k" Sep 05 07:48:52.581878 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.581853 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359-cert\") pod \"ingress-canary-rbg4k\" (UID: \"efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359\") " pod="openshift-ingress-canary/ingress-canary-rbg4k" Sep 05 07:48:52.587328 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.587303 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdrgh\" (UniqueName: \"kubernetes.io/projected/efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359-kube-api-access-wdrgh\") pod \"ingress-canary-rbg4k\" (UID: \"efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359\") " pod="openshift-ingress-canary/ingress-canary-rbg4k" Sep 05 07:48:52.622515 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.622491 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rbg4k" Sep 05 07:48:52.935174 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.935144 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rbg4k"] Sep 05 07:48:52.939565 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:48:52.939536 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa1fb29_89ac_4b0d_aa70_ab6e4a2ef359.slice/crio-9f1149035b23c013b8ae20612c1d97eb99015d799b668f1cf18d44694023531e WatchSource:0}: Error finding container 9f1149035b23c013b8ae20612c1d97eb99015d799b668f1cf18d44694023531e: Status 404 returned error can't find the container with id 9f1149035b23c013b8ae20612c1d97eb99015d799b668f1cf18d44694023531e Sep 05 07:48:52.982892 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.982864 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:52.983027 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:52.982932 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:52.983068 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:52.983019 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 07:48:52.983114 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:52.983070 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 07:48:52.983114 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:52.983083 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859f44b44c-q9ptx: secret "image-registry-tls" not found Sep 05 07:48:52.983114 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:52.983083 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls podName:332ec72e-6421-48a2-b0c0-def5d946f80c nodeName:}" failed. No retries permitted until 2026-09-05 07:48:53.983066986 +0000 UTC m=+36.592042092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls") pod "dns-default-7cs9n" (UID: "332ec72e-6421-48a2-b0c0-def5d946f80c") : secret "dns-default-metrics-tls" not found Sep 05 07:48:52.983233 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:52.983132 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls podName:a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc nodeName:}" failed. No retries permitted until 2026-09-05 07:48:53.983120956 +0000 UTC m=+36.592096058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls") pod "image-registry-859f44b44c-q9ptx" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc") : secret "image-registry-tls" not found Sep 05 07:48:53.170600 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.170567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rbg4k" event={"ID":"efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359","Type":"ContainerStarted","Data":"9f1149035b23c013b8ae20612c1d97eb99015d799b668f1cf18d44694023531e"} Sep 05 07:48:53.173663 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.173637 2576 generic.go:356] "Generic (PLEG): container finished" podID="e1d05ed4-423c-4619-89c1-3d4a00b8ea86" containerID="59f93e4f9ee061e5e94fdb9addfca6c3e0178421bd0b32a3e6187835ffa8238b" exitCode=0 Sep 05 07:48:53.173805 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.173674 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsknf" event={"ID":"e1d05ed4-423c-4619-89c1-3d4a00b8ea86","Type":"ContainerDied","Data":"59f93e4f9ee061e5e94fdb9addfca6c3e0178421bd0b32a3e6187835ffa8238b"} Sep 05 07:48:53.880941 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.880108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:48:53.880941 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.880107 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:48:53.880941 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.880175 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:48:53.883421 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.883240 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 05 07:48:53.883421 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.883254 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 05 07:48:53.883857 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.883627 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hp5x9\"" Sep 05 07:48:53.883857 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.883712 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g9xp6\"" Sep 05 07:48:53.883857 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.883747 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 05 07:48:53.883857 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.883825 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 05 07:48:53.989924 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.989873 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:53.990387 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:53.989941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:53.990387 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:53.990046 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 07:48:53.990387 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:53.990090 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 07:48:53.990387 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:53.990105 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859f44b44c-q9ptx: secret "image-registry-tls" not found Sep 05 07:48:53.990387 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:53.990111 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls podName:332ec72e-6421-48a2-b0c0-def5d946f80c nodeName:}" failed. No retries permitted until 2026-09-05 07:48:55.990095514 +0000 UTC m=+38.599070618 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls") pod "dns-default-7cs9n" (UID: "332ec72e-6421-48a2-b0c0-def5d946f80c") : secret "dns-default-metrics-tls" not found Sep 05 07:48:53.990387 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:53.990154 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls podName:a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc nodeName:}" failed. No retries permitted until 2026-09-05 07:48:55.990138453 +0000 UTC m=+38.599113553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls") pod "image-registry-859f44b44c-q9ptx" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc") : secret "image-registry-tls" not found Sep 05 07:48:54.180603 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:54.180509 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dsknf" event={"ID":"e1d05ed4-423c-4619-89c1-3d4a00b8ea86","Type":"ContainerStarted","Data":"de2fdd81e680f669f93c3ad81d7cfdd51d31027395691ea1bb69ff7b0e791def"} Sep 05 07:48:54.202842 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:54.202792 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dsknf" podStartSLOduration=8.278430643 podStartE2EDuration="36.202777374s" totalImagesPullingTime="27.924346731s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 07:48:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:48:54.201793574 +0000 UTC m=+36.810768697" watchObservedRunningTime="2026-09-05 07:48:54.202777374 +0000 UTC m=+36.811752496" Sep 05 07:48:56.007146 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:56.007103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:48:56.007563 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:56.007184 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:48:56.007563 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:56.007277 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 07:48:56.007563 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:56.007287 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 07:48:56.007563 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:56.007298 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859f44b44c-q9ptx: secret "image-registry-tls" not found Sep 05 07:48:56.007563 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:56.007342 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls podName:332ec72e-6421-48a2-b0c0-def5d946f80c nodeName:}" failed. No retries permitted until 2026-09-05 07:49:00.007329048 +0000 UTC m=+42.616304149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls") pod "dns-default-7cs9n" (UID: "332ec72e-6421-48a2-b0c0-def5d946f80c") : secret "dns-default-metrics-tls" not found Sep 05 07:48:56.007563 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:48:56.007356 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls podName:a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc nodeName:}" failed. No retries permitted until 2026-09-05 07:49:00.007349946 +0000 UTC m=+42.616325047 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls") pod "image-registry-859f44b44c-q9ptx" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc") : secret "image-registry-tls" not found Sep 05 07:48:56.187331 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:56.187296 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rbg4k" event={"ID":"efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359","Type":"ContainerStarted","Data":"bcf13088bbc701061bc32c7f182c618f9503e899c3373652df5c9c87ebd1f8c6"} Sep 05 07:48:56.202829 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:48:56.202781 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rbg4k" podStartSLOduration=1.33669349 podStartE2EDuration="4.202766637s" totalImagesPullingTime="2.866073147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:48:56.201739532 +0000 UTC m=+38.810714655" watchObservedRunningTime="2026-09-05 07:48:56.202766637 +0000 UTC m=+38.811741820" Sep 05 07:49:00.040536 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:00.040485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:49:00.040996 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:00.040573 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:49:00.040996 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:00.040629 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 07:49:00.040996 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:00.040701 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls podName:332ec72e-6421-48a2-b0c0-def5d946f80c nodeName:}" failed. No retries permitted until 2026-09-05 07:49:08.040685379 +0000 UTC m=+50.649660481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls") pod "dns-default-7cs9n" (UID: "332ec72e-6421-48a2-b0c0-def5d946f80c") : secret "dns-default-metrics-tls" not found Sep 05 07:49:00.040996 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:00.040725 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 07:49:00.040996 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:00.040739 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859f44b44c-q9ptx: secret "image-registry-tls" not found Sep 05 07:49:00.040996 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:00.040794 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls podName:a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc nodeName:}" failed. No retries permitted until 2026-09-05 07:49:08.040777917 +0000 UTC m=+50.649753032 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls") pod "image-registry-859f44b44c-q9ptx" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc") : secret "image-registry-tls" not found Sep 05 07:49:00.973248 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:00.973198 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-8z556"] Sep 05 07:49:01.016988 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.016957 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-hhvm4"] Sep 05 07:49:01.017167 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.017108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-8z556" Sep 05 07:49:01.019157 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.019136 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vcvp2\"" Sep 05 07:49:01.030307 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.030287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-8z556"] Sep 05 07:49:01.030402 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.030322 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-hhvm4"] Sep 05 07:49:01.030441 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.030403 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.032466 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.032426 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 05 07:49:01.032466 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.032435 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dszwq\"" Sep 05 07:49:01.032629 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.032455 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 05 07:49:01.032629 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.032468 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 05 07:49:01.032629 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.032591 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 05 07:49:01.038181 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.038157 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 05 07:49:01.076396 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.076367 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg"] Sep 05 07:49:01.093145 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.093117 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7"] Sep 05 07:49:01.093289 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.093273 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:01.095345 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.095322 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 05 07:49:01.095502 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.095476 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 05 07:49:01.095502 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.095491 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 05 07:49:01.095727 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.095712 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xsf7v\"" Sep 05 07:49:01.096020 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.096003 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 05 07:49:01.111900 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.111878 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg"] Sep 05 07:49:01.111970 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.111907 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7"] Sep 05 07:49:01.112056 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.111983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" Sep 05 07:49:01.114092 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.114058 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 05 07:49:01.114092 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.114063 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-d8m7s\"" Sep 05 07:49:01.114092 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.114085 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 05 07:49:01.114387 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.114375 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 05 07:49:01.114426 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.114409 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 05 07:49:01.149587 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.149560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3da53738-9ae4-473e-b55a-d7379875a4a6-trusted-ca\") pod \"console-operator-85b454874b-hhvm4\" (UID: \"3da53738-9ae4-473e-b55a-d7379875a4a6\") " pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.149753 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.149594 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mvvx\" (UniqueName: \"kubernetes.io/projected/62fd2b86-c2f9-482d-bd36-5e2db6bacce4-kube-api-access-5mvvx\") pod \"network-check-source-685456655f-8z556\" (UID: \"62fd2b86-c2f9-482d-bd36-5e2db6bacce4\") " pod="openshift-network-diagnostics/network-check-source-685456655f-8z556" Sep 05 07:49:01.149753 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.149613 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt8rd\" (UniqueName: \"kubernetes.io/projected/3da53738-9ae4-473e-b55a-d7379875a4a6-kube-api-access-jt8rd\") pod \"console-operator-85b454874b-hhvm4\" (UID: \"3da53738-9ae4-473e-b55a-d7379875a4a6\") " pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.149753 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.149651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3da53738-9ae4-473e-b55a-d7379875a4a6-config\") pod \"console-operator-85b454874b-hhvm4\" (UID: \"3da53738-9ae4-473e-b55a-d7379875a4a6\") " pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.149753 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.149724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3da53738-9ae4-473e-b55a-d7379875a4a6-serving-cert\") pod \"console-operator-85b454874b-hhvm4\" (UID: \"3da53738-9ae4-473e-b55a-d7379875a4a6\") " pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.250965 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.250868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:01.250965 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.250922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz75p\" (UniqueName: \"kubernetes.io/projected/c0bd8583-070e-4b16-96bf-3780bbb5497f-kube-api-access-nz75p\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:01.251198 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.250967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0902955-102e-4493-83f5-a427d9f7c1e8-serving-cert\") pod \"service-ca-operator-859f94c4b-dp8m7\" (UID: \"e0902955-102e-4493-83f5-a427d9f7c1e8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" Sep 05 07:49:01.251198 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.250989 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6fnn\" (UniqueName: \"kubernetes.io/projected/e0902955-102e-4493-83f5-a427d9f7c1e8-kube-api-access-z6fnn\") pod \"service-ca-operator-859f94c4b-dp8m7\" (UID: \"e0902955-102e-4493-83f5-a427d9f7c1e8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" Sep 05 07:49:01.251198 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.251017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3da53738-9ae4-473e-b55a-d7379875a4a6-trusted-ca\") pod \"console-operator-85b454874b-hhvm4\" (UID: \"3da53738-9ae4-473e-b55a-d7379875a4a6\") " pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.251198 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.251054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvvx\" (UniqueName: \"kubernetes.io/projected/62fd2b86-c2f9-482d-bd36-5e2db6bacce4-kube-api-access-5mvvx\") pod \"network-check-source-685456655f-8z556\" (UID: \"62fd2b86-c2f9-482d-bd36-5e2db6bacce4\") " pod="openshift-network-diagnostics/network-check-source-685456655f-8z556" Sep 05 07:49:01.251198 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.251079 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt8rd\" (UniqueName: \"kubernetes.io/projected/3da53738-9ae4-473e-b55a-d7379875a4a6-kube-api-access-jt8rd\") pod \"console-operator-85b454874b-hhvm4\" (UID: \"3da53738-9ae4-473e-b55a-d7379875a4a6\") " pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.251198 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.251105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0902955-102e-4493-83f5-a427d9f7c1e8-config\") pod \"service-ca-operator-859f94c4b-dp8m7\" (UID: \"e0902955-102e-4493-83f5-a427d9f7c1e8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" Sep 05 07:49:01.251198 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.251168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3da53738-9ae4-473e-b55a-d7379875a4a6-config\") pod \"console-operator-85b454874b-hhvm4\" (UID: \"3da53738-9ae4-473e-b55a-d7379875a4a6\") " pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.251198 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.251192 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c0bd8583-070e-4b16-96bf-3780bbb5497f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:01.251625 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.251282 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3da53738-9ae4-473e-b55a-d7379875a4a6-serving-cert\") pod \"console-operator-85b454874b-hhvm4\" (UID: \"3da53738-9ae4-473e-b55a-d7379875a4a6\") " pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.252002 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.251979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3da53738-9ae4-473e-b55a-d7379875a4a6-config\") pod \"console-operator-85b454874b-hhvm4\" (UID: \"3da53738-9ae4-473e-b55a-d7379875a4a6\") " pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.252066 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.252010 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3da53738-9ae4-473e-b55a-d7379875a4a6-trusted-ca\") pod \"console-operator-85b454874b-hhvm4\" (UID: \"3da53738-9ae4-473e-b55a-d7379875a4a6\") " pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.255150 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.255125 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3da53738-9ae4-473e-b55a-d7379875a4a6-serving-cert\") pod \"console-operator-85b454874b-hhvm4\" (UID: \"3da53738-9ae4-473e-b55a-d7379875a4a6\") " pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.259976 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.259954 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt8rd\" (UniqueName: \"kubernetes.io/projected/3da53738-9ae4-473e-b55a-d7379875a4a6-kube-api-access-jt8rd\") pod \"console-operator-85b454874b-hhvm4\" (UID: \"3da53738-9ae4-473e-b55a-d7379875a4a6\") " pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.260409 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.260389 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mvvx\" (UniqueName: \"kubernetes.io/projected/62fd2b86-c2f9-482d-bd36-5e2db6bacce4-kube-api-access-5mvvx\") pod \"network-check-source-685456655f-8z556\" (UID: \"62fd2b86-c2f9-482d-bd36-5e2db6bacce4\") " pod="openshift-network-diagnostics/network-check-source-685456655f-8z556" Sep 05 07:49:01.325759 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.325722 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-8z556" Sep 05 07:49:01.338981 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.338943 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:01.351936 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.351623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0902955-102e-4493-83f5-a427d9f7c1e8-config\") pod \"service-ca-operator-859f94c4b-dp8m7\" (UID: \"e0902955-102e-4493-83f5-a427d9f7c1e8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" Sep 05 07:49:01.351936 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.351679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c0bd8583-070e-4b16-96bf-3780bbb5497f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:01.351936 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.351738 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:01.351936 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.351761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz75p\" (UniqueName: \"kubernetes.io/projected/c0bd8583-070e-4b16-96bf-3780bbb5497f-kube-api-access-nz75p\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:01.351936 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.351777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0902955-102e-4493-83f5-a427d9f7c1e8-serving-cert\") pod \"service-ca-operator-859f94c4b-dp8m7\" (UID: \"e0902955-102e-4493-83f5-a427d9f7c1e8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" Sep 05 07:49:01.351936 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.351801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6fnn\" (UniqueName: \"kubernetes.io/projected/e0902955-102e-4493-83f5-a427d9f7c1e8-kube-api-access-z6fnn\") pod \"service-ca-operator-859f94c4b-dp8m7\" (UID: \"e0902955-102e-4493-83f5-a427d9f7c1e8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" Sep 05 07:49:01.351936 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:01.351883 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 05 07:49:01.352280 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:01.352058 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls podName:c0bd8583-070e-4b16-96bf-3780bbb5497f nodeName:}" failed. No retries permitted until 2026-09-05 07:49:01.852039132 +0000 UTC m=+44.461014248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-npnpg" (UID: "c0bd8583-070e-4b16-96bf-3780bbb5497f") : secret "cluster-monitoring-operator-tls" not found Sep 05 07:49:01.352280 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.352262 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e0902955-102e-4493-83f5-a427d9f7c1e8-config\") pod \"service-ca-operator-859f94c4b-dp8m7\" (UID: \"e0902955-102e-4493-83f5-a427d9f7c1e8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" Sep 05 07:49:01.352607 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.352575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c0bd8583-070e-4b16-96bf-3780bbb5497f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:01.354342 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.354308 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e0902955-102e-4493-83f5-a427d9f7c1e8-serving-cert\") pod \"service-ca-operator-859f94c4b-dp8m7\" (UID: \"e0902955-102e-4493-83f5-a427d9f7c1e8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" Sep 05 07:49:01.362482 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.362441 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6fnn\" (UniqueName: \"kubernetes.io/projected/e0902955-102e-4493-83f5-a427d9f7c1e8-kube-api-access-z6fnn\") pod \"service-ca-operator-859f94c4b-dp8m7\" (UID: \"e0902955-102e-4493-83f5-a427d9f7c1e8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" Sep 05 07:49:01.362982 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.362957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz75p\" (UniqueName: \"kubernetes.io/projected/c0bd8583-070e-4b16-96bf-3780bbb5497f-kube-api-access-nz75p\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:01.422959 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.422929 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" Sep 05 07:49:01.620814 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.620779 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-8z556"] Sep 05 07:49:01.623975 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:01.623944 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fd2b86_c2f9_482d_bd36_5e2db6bacce4.slice/crio-8530a672ef868b789fa8aef6d001ca3f5f31c4402d51d4c48275fd4cbe725d61 WatchSource:0}: Error finding container 8530a672ef868b789fa8aef6d001ca3f5f31c4402d51d4c48275fd4cbe725d61: Status 404 returned error can't find the container with id 8530a672ef868b789fa8aef6d001ca3f5f31c4402d51d4c48275fd4cbe725d61 Sep 05 07:49:01.657981 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.657951 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-hhvm4"] Sep 05 07:49:01.661096 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:01.661072 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da53738_9ae4_473e_b55a_d7379875a4a6.slice/crio-7cecd8429840251b7fa513419dbbc6e4cf85d5a8d3b37a218df66fd9d1194d1b WatchSource:0}: Error finding container 7cecd8429840251b7fa513419dbbc6e4cf85d5a8d3b37a218df66fd9d1194d1b: Status 404 returned error can't find the container with id 7cecd8429840251b7fa513419dbbc6e4cf85d5a8d3b37a218df66fd9d1194d1b Sep 05 07:49:01.718394 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.718363 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7"] Sep 05 07:49:01.721699 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:01.721664 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0902955_102e_4493_83f5_a427d9f7c1e8.slice/crio-d715085e4d08df256d3ceeddfc305143f4dc5bad32686776f816dd467f7b4738 WatchSource:0}: Error finding container d715085e4d08df256d3ceeddfc305143f4dc5bad32686776f816dd467f7b4738: Status 404 returned error can't find the container with id d715085e4d08df256d3ceeddfc305143f4dc5bad32686776f816dd467f7b4738 Sep 05 07:49:01.856570 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:01.856522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:01.856753 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:01.856678 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 05 07:49:01.856753 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:01.856747 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls podName:c0bd8583-070e-4b16-96bf-3780bbb5497f nodeName:}" failed. No retries permitted until 2026-09-05 07:49:02.85673157 +0000 UTC m=+45.465706672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-npnpg" (UID: "c0bd8583-070e-4b16-96bf-3780bbb5497f") : secret "cluster-monitoring-operator-tls" not found Sep 05 07:49:02.203259 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:02.203201 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" event={"ID":"3da53738-9ae4-473e-b55a-d7379875a4a6","Type":"ContainerStarted","Data":"7cecd8429840251b7fa513419dbbc6e4cf85d5a8d3b37a218df66fd9d1194d1b"} Sep 05 07:49:02.204828 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:02.204799 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-8z556" event={"ID":"62fd2b86-c2f9-482d-bd36-5e2db6bacce4","Type":"ContainerStarted","Data":"8530a672ef868b789fa8aef6d001ca3f5f31c4402d51d4c48275fd4cbe725d61"} Sep 05 07:49:02.206202 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:02.206171 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" event={"ID":"e0902955-102e-4493-83f5-a427d9f7c1e8","Type":"ContainerStarted","Data":"d715085e4d08df256d3ceeddfc305143f4dc5bad32686776f816dd467f7b4738"} Sep 05 07:49:02.866253 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:02.865592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:02.866253 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:02.865810 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 05 07:49:02.866253 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:02.865881 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls podName:c0bd8583-070e-4b16-96bf-3780bbb5497f nodeName:}" failed. No retries permitted until 2026-09-05 07:49:04.865859115 +0000 UTC m=+47.474834221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-npnpg" (UID: "c0bd8583-070e-4b16-96bf-3780bbb5497f") : secret "cluster-monitoring-operator-tls" not found Sep 05 07:49:04.884937 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:04.884890 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:04.885415 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:04.885076 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 05 07:49:04.885415 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:04.885159 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls podName:c0bd8583-070e-4b16-96bf-3780bbb5497f nodeName:}" failed. No retries permitted until 2026-09-05 07:49:08.885137929 +0000 UTC m=+51.494113045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-npnpg" (UID: "c0bd8583-070e-4b16-96bf-3780bbb5497f") : secret "cluster-monitoring-operator-tls" not found Sep 05 07:49:06.364058 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:06.364019 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m9n7b"] Sep 05 07:49:06.387915 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:06.387879 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m9n7b"] Sep 05 07:49:06.388027 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:06.388014 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m9n7b" Sep 05 07:49:06.390452 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:06.390428 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 05 07:49:06.391098 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:06.391083 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 05 07:49:06.391194 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:06.391135 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2cw2n\"" Sep 05 07:49:06.400546 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:06.400527 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsbp6\" (UniqueName: \"kubernetes.io/projected/6655c29a-17e5-4402-b6b1-4550f3ac99dc-kube-api-access-gsbp6\") pod \"migrator-fdbb8fdff-m9n7b\" (UID: \"6655c29a-17e5-4402-b6b1-4550f3ac99dc\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m9n7b" Sep 05 07:49:06.501791 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:06.501702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsbp6\" (UniqueName: \"kubernetes.io/projected/6655c29a-17e5-4402-b6b1-4550f3ac99dc-kube-api-access-gsbp6\") pod \"migrator-fdbb8fdff-m9n7b\" (UID: \"6655c29a-17e5-4402-b6b1-4550f3ac99dc\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m9n7b" Sep 05 07:49:06.509167 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:06.509128 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsbp6\" (UniqueName: \"kubernetes.io/projected/6655c29a-17e5-4402-b6b1-4550f3ac99dc-kube-api-access-gsbp6\") pod \"migrator-fdbb8fdff-m9n7b\" (UID: \"6655c29a-17e5-4402-b6b1-4550f3ac99dc\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m9n7b" Sep 05 07:49:06.701009 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:06.700973 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m9n7b" Sep 05 07:49:06.989955 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:06.989908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m9n7b"] Sep 05 07:49:06.994843 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:06.994818 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6655c29a_17e5_4402_b6b1_4550f3ac99dc.slice/crio-96a2dd019dea6b08fa8bb3e4bd64f2998c0ae0a61a7723aa6f82a4fb66945755 WatchSource:0}: Error finding container 96a2dd019dea6b08fa8bb3e4bd64f2998c0ae0a61a7723aa6f82a4fb66945755: Status 404 returned error can't find the container with id 96a2dd019dea6b08fa8bb3e4bd64f2998c0ae0a61a7723aa6f82a4fb66945755 Sep 05 07:49:07.015896 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.015833 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-cf2bt"] Sep 05 07:49:07.033511 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.033480 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cf2bt"] Sep 05 07:49:07.033640 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.033597 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.036269 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.035772 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 05 07:49:07.036269 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.036092 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gx2xl\"" Sep 05 07:49:07.036269 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.036103 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 05 07:49:07.036269 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.036173 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 05 07:49:07.036269 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.036178 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 05 07:49:07.105081 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.105050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.105271 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.105112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2bc7982-908a-4621-b8d8-962a902ff34d-data-volume\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.105271 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.105161 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a2bc7982-908a-4621-b8d8-962a902ff34d-crio-socket\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.105271 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.105180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2bc7982-908a-4621-b8d8-962a902ff34d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.105380 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.105323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkg5x\" (UniqueName: \"kubernetes.io/projected/a2bc7982-908a-4621-b8d8-962a902ff34d-kube-api-access-mkg5x\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.206348 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.206311 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a2bc7982-908a-4621-b8d8-962a902ff34d-crio-socket\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.206348 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.206350 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2bc7982-908a-4621-b8d8-962a902ff34d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.206533 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:07.206428 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 05 07:49:07.206533 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:07.206503 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2bc7982-908a-4621-b8d8-962a902ff34d-kube-rbac-proxy-cm podName:a2bc7982-908a-4621-b8d8-962a902ff34d nodeName:}" failed. No retries permitted until 2026-09-05 07:49:07.706484357 +0000 UTC m=+50.315459480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a2bc7982-908a-4621-b8d8-962a902ff34d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cf2bt" (UID: "a2bc7982-908a-4621-b8d8-962a902ff34d") : configmap "kube-rbac-proxy" not found Sep 05 07:49:07.206533 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.206520 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkg5x\" (UniqueName: \"kubernetes.io/projected/a2bc7982-908a-4621-b8d8-962a902ff34d-kube-api-access-mkg5x\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.206659 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.206550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.206659 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.206622 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2bc7982-908a-4621-b8d8-962a902ff34d-data-volume\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.206659 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:07.206631 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 07:49:07.206659 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.206549 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a2bc7982-908a-4621-b8d8-962a902ff34d-crio-socket\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.206804 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:07.206672 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls podName:a2bc7982-908a-4621-b8d8-962a902ff34d nodeName:}" failed. No retries permitted until 2026-09-05 07:49:07.706657754 +0000 UTC m=+50.315632857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cf2bt" (UID: "a2bc7982-908a-4621-b8d8-962a902ff34d") : secret "insights-runtime-extractor-tls" not found Sep 05 07:49:07.206942 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.206926 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2bc7982-908a-4621-b8d8-962a902ff34d-data-volume\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.214135 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.214112 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkg5x\" (UniqueName: \"kubernetes.io/projected/a2bc7982-908a-4621-b8d8-962a902ff34d-kube-api-access-mkg5x\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.221145 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.221122 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/0.log" Sep 05 07:49:07.221291 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.221167 2576 generic.go:356] "Generic (PLEG): container finished" podID="3da53738-9ae4-473e-b55a-d7379875a4a6" containerID="5c3bc1a852e233e6bf9661ee03448fa4f8e7fd015cf0fa75ec4f04a90223e500" exitCode=255 Sep 05 07:49:07.221291 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.221252 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" event={"ID":"3da53738-9ae4-473e-b55a-d7379875a4a6","Type":"ContainerDied","Data":"5c3bc1a852e233e6bf9661ee03448fa4f8e7fd015cf0fa75ec4f04a90223e500"} Sep 05 07:49:07.221550 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.221531 2576 scope.go:117] "RemoveContainer" containerID="5c3bc1a852e233e6bf9661ee03448fa4f8e7fd015cf0fa75ec4f04a90223e500" Sep 05 07:49:07.222938 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.222912 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-8z556" event={"ID":"62fd2b86-c2f9-482d-bd36-5e2db6bacce4","Type":"ContainerStarted","Data":"b0995782d2cd2f67fb60ef9a77a99a52a1cc0dd6fe598ef13f93dc568643829f"} Sep 05 07:49:07.224683 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.224659 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" event={"ID":"e0902955-102e-4493-83f5-a427d9f7c1e8","Type":"ContainerStarted","Data":"a9773b22e45baac0cec37a7da48bd03356729b7b250310c8101b299dcfe932da"} Sep 05 07:49:07.225881 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.225861 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m9n7b" event={"ID":"6655c29a-17e5-4402-b6b1-4550f3ac99dc","Type":"ContainerStarted","Data":"96a2dd019dea6b08fa8bb3e4bd64f2998c0ae0a61a7723aa6f82a4fb66945755"} Sep 05 07:49:07.248929 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.248885 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-8z556" podStartSLOduration=2.741975534 podStartE2EDuration="7.248872568s" totalImagesPullingTime="4.506897034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:07.248592924 +0000 UTC m=+49.857568048" watchObservedRunningTime="2026-09-05 07:49:07.248872568 +0000 UTC m=+49.857847690" Sep 05 07:49:07.264736 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.264397 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" podStartSLOduration=1.847673555 podStartE2EDuration="6.264383582s" totalImagesPullingTime="4.416710027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:07.262789817 +0000 UTC m=+49.871764940" watchObservedRunningTime="2026-09-05 07:49:07.264383582 +0000 UTC m=+49.873358706" Sep 05 07:49:07.711808 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.711773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2bc7982-908a-4621-b8d8-962a902ff34d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.712314 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:07.711825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:07.712314 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:07.711917 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 05 07:49:07.712314 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:07.711931 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 07:49:07.712314 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:07.711984 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls podName:a2bc7982-908a-4621-b8d8-962a902ff34d nodeName:}" failed. No retries permitted until 2026-09-05 07:49:08.711970542 +0000 UTC m=+51.320945643 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cf2bt" (UID: "a2bc7982-908a-4621-b8d8-962a902ff34d") : secret "insights-runtime-extractor-tls" not found Sep 05 07:49:07.712314 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:07.711997 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a2bc7982-908a-4621-b8d8-962a902ff34d-kube-rbac-proxy-cm podName:a2bc7982-908a-4621-b8d8-962a902ff34d nodeName:}" failed. No retries permitted until 2026-09-05 07:49:08.711990135 +0000 UTC m=+51.320965236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a2bc7982-908a-4621-b8d8-962a902ff34d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-cf2bt" (UID: "a2bc7982-908a-4621-b8d8-962a902ff34d") : configmap "kube-rbac-proxy" not found Sep 05 07:49:08.116662 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:08.116614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:49:08.116871 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:08.116764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:49:08.116871 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:08.116771 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 05 07:49:08.116871 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:08.116794 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859f44b44c-q9ptx: secret "image-registry-tls" not found Sep 05 07:49:08.116871 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:08.116854 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls podName:a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc nodeName:}" failed. No retries permitted until 2026-09-05 07:49:24.116837594 +0000 UTC m=+66.725812696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls") pod "image-registry-859f44b44c-q9ptx" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc") : secret "image-registry-tls" not found Sep 05 07:49:08.117063 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:08.116891 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 05 07:49:08.117063 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:08.116948 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls podName:332ec72e-6421-48a2-b0c0-def5d946f80c nodeName:}" failed. No retries permitted until 2026-09-05 07:49:24.11693237 +0000 UTC m=+66.725907489 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls") pod "dns-default-7cs9n" (UID: "332ec72e-6421-48a2-b0c0-def5d946f80c") : secret "dns-default-metrics-tls" not found Sep 05 07:49:08.231957 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:08.231928 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 07:49:08.232452 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:08.232432 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/0.log" Sep 05 07:49:08.232585 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:08.232471 2576 generic.go:356] "Generic (PLEG): container finished" podID="3da53738-9ae4-473e-b55a-d7379875a4a6" containerID="324d6cc8cc31fcb68fdb9ac60dbd1b8eeeeaaf86aee3e172c759989ca340c84b" exitCode=255 Sep 05 07:49:08.232585 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:08.232507 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" event={"ID":"3da53738-9ae4-473e-b55a-d7379875a4a6","Type":"ContainerDied","Data":"324d6cc8cc31fcb68fdb9ac60dbd1b8eeeeaaf86aee3e172c759989ca340c84b"} Sep 05 07:49:08.249330 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:08.248665 2576 scope.go:117] "RemoveContainer" containerID="5c3bc1a852e233e6bf9661ee03448fa4f8e7fd015cf0fa75ec4f04a90223e500" Sep 05 07:49:08.249330 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:08.248817 2576 scope.go:117] "RemoveContainer" containerID="324d6cc8cc31fcb68fdb9ac60dbd1b8eeeeaaf86aee3e172c759989ca340c84b" Sep 05 07:49:08.249330 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:08.249070 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-hhvm4_openshift-console-operator(3da53738-9ae4-473e-b55a-d7379875a4a6)\"" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" podUID="3da53738-9ae4-473e-b55a-d7379875a4a6" Sep 05 07:49:08.721155 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:08.721126 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2bc7982-908a-4621-b8d8-962a902ff34d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:08.721480 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:08.721186 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:08.721480 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:08.721331 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 07:49:08.721480 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:08.721390 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls podName:a2bc7982-908a-4621-b8d8-962a902ff34d nodeName:}" failed. No retries permitted until 2026-09-05 07:49:10.721373999 +0000 UTC m=+53.330349102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cf2bt" (UID: "a2bc7982-908a-4621-b8d8-962a902ff34d") : secret "insights-runtime-extractor-tls" not found Sep 05 07:49:08.721818 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:08.721794 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a2bc7982-908a-4621-b8d8-962a902ff34d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:08.923070 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:08.923027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:08.923322 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:08.923160 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 05 07:49:08.923322 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:08.923233 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls podName:c0bd8583-070e-4b16-96bf-3780bbb5497f nodeName:}" failed. No retries permitted until 2026-09-05 07:49:16.923198633 +0000 UTC m=+59.532173735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-npnpg" (UID: "c0bd8583-070e-4b16-96bf-3780bbb5497f") : secret "cluster-monitoring-operator-tls" not found Sep 05 07:49:09.165198 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.165171 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4bmbd_a6bf702d-539a-4164-9e92-18a93205e30f/dns-node-resolver/0.log" Sep 05 07:49:09.237168 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.237140 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 07:49:09.237552 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.237533 2576 scope.go:117] "RemoveContainer" containerID="324d6cc8cc31fcb68fdb9ac60dbd1b8eeeeaaf86aee3e172c759989ca340c84b" Sep 05 07:49:09.237733 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:09.237707 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-hhvm4_openshift-console-operator(3da53738-9ae4-473e-b55a-d7379875a4a6)\"" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" podUID="3da53738-9ae4-473e-b55a-d7379875a4a6" Sep 05 07:49:09.239169 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.239148 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m9n7b" event={"ID":"6655c29a-17e5-4402-b6b1-4550f3ac99dc","Type":"ContainerStarted","Data":"d40353964f00e8e7849c3a2ff074e6876e5575d76fa9035c5925733284f41bcf"} Sep 05 07:49:09.239273 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.239192 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m9n7b" event={"ID":"6655c29a-17e5-4402-b6b1-4550f3ac99dc","Type":"ContainerStarted","Data":"2803d9ddce9f2795c6c84fe16f5b230d9bc7d35aa98060e62db15b558286eb11"} Sep 05 07:49:09.269661 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.269616 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-m9n7b" podStartSLOduration=1.722358581 podStartE2EDuration="3.26960232s" totalImagesPullingTime="1.547243739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:09.269080919 +0000 UTC m=+51.878056044" watchObservedRunningTime="2026-09-05 07:49:09.26960232 +0000 UTC m=+51.878577486" Sep 05 07:49:09.534767 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.534688 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-pbhgn"] Sep 05 07:49:09.556917 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.556887 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-pbhgn"] Sep 05 07:49:09.557066 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.556983 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-pbhgn" Sep 05 07:49:09.559989 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.559966 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 05 07:49:09.560300 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.560283 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 05 07:49:09.560417 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.560397 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-w5bpw\"" Sep 05 07:49:09.560744 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.560727 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 05 07:49:09.561009 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.560993 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 05 07:49:09.628282 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.628239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9e355876-3e90-4b00-bfa5-e2bf0ff2d655-signing-cabundle\") pod \"service-ca-75d99c484-pbhgn\" (UID: \"9e355876-3e90-4b00-bfa5-e2bf0ff2d655\") " pod="openshift-service-ca/service-ca-75d99c484-pbhgn" Sep 05 07:49:09.628445 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.628375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9e355876-3e90-4b00-bfa5-e2bf0ff2d655-signing-key\") pod \"service-ca-75d99c484-pbhgn\" (UID: \"9e355876-3e90-4b00-bfa5-e2bf0ff2d655\") " pod="openshift-service-ca/service-ca-75d99c484-pbhgn" Sep 05 07:49:09.628445 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.628409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qpm8\" (UniqueName: \"kubernetes.io/projected/9e355876-3e90-4b00-bfa5-e2bf0ff2d655-kube-api-access-2qpm8\") pod \"service-ca-75d99c484-pbhgn\" (UID: \"9e355876-3e90-4b00-bfa5-e2bf0ff2d655\") " pod="openshift-service-ca/service-ca-75d99c484-pbhgn" Sep 05 07:49:09.728894 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.728849 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9e355876-3e90-4b00-bfa5-e2bf0ff2d655-signing-cabundle\") pod \"service-ca-75d99c484-pbhgn\" (UID: \"9e355876-3e90-4b00-bfa5-e2bf0ff2d655\") " pod="openshift-service-ca/service-ca-75d99c484-pbhgn" Sep 05 07:49:09.729358 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.728967 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9e355876-3e90-4b00-bfa5-e2bf0ff2d655-signing-key\") pod \"service-ca-75d99c484-pbhgn\" (UID: \"9e355876-3e90-4b00-bfa5-e2bf0ff2d655\") " pod="openshift-service-ca/service-ca-75d99c484-pbhgn" Sep 05 07:49:09.729358 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.728995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qpm8\" (UniqueName: \"kubernetes.io/projected/9e355876-3e90-4b00-bfa5-e2bf0ff2d655-kube-api-access-2qpm8\") pod \"service-ca-75d99c484-pbhgn\" (UID: \"9e355876-3e90-4b00-bfa5-e2bf0ff2d655\") " pod="openshift-service-ca/service-ca-75d99c484-pbhgn" Sep 05 07:49:09.729687 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.729658 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9e355876-3e90-4b00-bfa5-e2bf0ff2d655-signing-cabundle\") pod \"service-ca-75d99c484-pbhgn\" (UID: \"9e355876-3e90-4b00-bfa5-e2bf0ff2d655\") " pod="openshift-service-ca/service-ca-75d99c484-pbhgn" Sep 05 07:49:09.731544 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.731517 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9e355876-3e90-4b00-bfa5-e2bf0ff2d655-signing-key\") pod \"service-ca-75d99c484-pbhgn\" (UID: \"9e355876-3e90-4b00-bfa5-e2bf0ff2d655\") " pod="openshift-service-ca/service-ca-75d99c484-pbhgn" Sep 05 07:49:09.737632 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.737611 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qpm8\" (UniqueName: \"kubernetes.io/projected/9e355876-3e90-4b00-bfa5-e2bf0ff2d655-kube-api-access-2qpm8\") pod \"service-ca-75d99c484-pbhgn\" (UID: \"9e355876-3e90-4b00-bfa5-e2bf0ff2d655\") " pod="openshift-service-ca/service-ca-75d99c484-pbhgn" Sep 05 07:49:09.869530 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:09.869490 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-pbhgn" Sep 05 07:49:10.162818 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:10.162741 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-pbhgn"] Sep 05 07:49:10.166341 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:10.166322 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-brf5d_13432308-4a00-44e2-babf-6daaff232a0a/node-ca/0.log" Sep 05 07:49:10.166684 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:10.166648 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e355876_3e90_4b00_bfa5_e2bf0ff2d655.slice/crio-1ac02ce3693459f7f53b8f6dae1e7be5708b8d15b80944c3a0d1601dc748d021 WatchSource:0}: Error finding container 1ac02ce3693459f7f53b8f6dae1e7be5708b8d15b80944c3a0d1601dc748d021: Status 404 returned error can't find the container with id 1ac02ce3693459f7f53b8f6dae1e7be5708b8d15b80944c3a0d1601dc748d021 Sep 05 07:49:10.243931 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:10.243898 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-pbhgn" event={"ID":"9e355876-3e90-4b00-bfa5-e2bf0ff2d655","Type":"ContainerStarted","Data":"1ac02ce3693459f7f53b8f6dae1e7be5708b8d15b80944c3a0d1601dc748d021"} Sep 05 07:49:10.738561 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:10.738525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:10.738922 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:10.738643 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 07:49:10.738922 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:10.738701 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls podName:a2bc7982-908a-4621-b8d8-962a902ff34d nodeName:}" failed. No retries permitted until 2026-09-05 07:49:14.738686139 +0000 UTC m=+57.347661240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cf2bt" (UID: "a2bc7982-908a-4621-b8d8-962a902ff34d") : secret "insights-runtime-extractor-tls" not found Sep 05 07:49:11.167574 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:11.167547 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rbg4k_efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359/serve-healthcheck-canary/0.log" Sep 05 07:49:11.248418 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:11.248381 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-pbhgn" event={"ID":"9e355876-3e90-4b00-bfa5-e2bf0ff2d655","Type":"ContainerStarted","Data":"48605a6b7a79f2ebd95142ca96bed26465e5f937bcc2a5288c276b3bc53ab443"} Sep 05 07:49:11.265620 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:11.265571 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-pbhgn" podStartSLOduration=2.265555643 podStartE2EDuration="2.265555643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:11.264935294 +0000 UTC m=+53.873910417" watchObservedRunningTime="2026-09-05 07:49:11.265555643 +0000 UTC m=+53.874530766" Sep 05 07:49:11.339846 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:11.339804 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:11.339993 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:11.339870 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:11.340248 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:11.340234 2576 scope.go:117] "RemoveContainer" containerID="324d6cc8cc31fcb68fdb9ac60dbd1b8eeeeaaf86aee3e172c759989ca340c84b" Sep 05 07:49:11.340402 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:11.340386 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-hhvm4_openshift-console-operator(3da53738-9ae4-473e-b55a-d7379875a4a6)\"" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" podUID="3da53738-9ae4-473e-b55a-d7379875a4a6" Sep 05 07:49:11.367023 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:11.366986 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-m9n7b_6655c29a-17e5-4402-b6b1-4550f3ac99dc/migrator/0.log" Sep 05 07:49:11.576748 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:11.576718 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-m9n7b_6655c29a-17e5-4402-b6b1-4550f3ac99dc/graceful-termination/0.log" Sep 05 07:49:14.777369 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:14.777315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:14.777776 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:14.777499 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 05 07:49:14.777776 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:14.777586 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls podName:a2bc7982-908a-4621-b8d8-962a902ff34d nodeName:}" failed. No retries permitted until 2026-09-05 07:49:22.777564696 +0000 UTC m=+65.386539803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-cf2bt" (UID: "a2bc7982-908a-4621-b8d8-962a902ff34d") : secret "insights-runtime-extractor-tls" not found Sep 05 07:49:16.996672 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:16.996617 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:16.997170 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:16.996790 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 05 07:49:16.997170 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:49:16.996880 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls podName:c0bd8583-070e-4b16-96bf-3780bbb5497f nodeName:}" failed. No retries permitted until 2026-09-05 07:49:32.996858585 +0000 UTC m=+75.605833685 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-npnpg" (UID: "c0bd8583-070e-4b16-96bf-3780bbb5497f") : secret "cluster-monitoring-operator-tls" not found Sep 05 07:49:19.168101 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:19.168066 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zbn86" Sep 05 07:49:21.880898 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:21.880866 2576 scope.go:117] "RemoveContainer" containerID="324d6cc8cc31fcb68fdb9ac60dbd1b8eeeeaaf86aee3e172c759989ca340c84b" Sep 05 07:49:22.289874 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:22.289772 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 07:49:22.290042 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:22.289872 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" event={"ID":"3da53738-9ae4-473e-b55a-d7379875a4a6","Type":"ContainerStarted","Data":"5dd3cb763b7eac91a7bf2dfba91a639cda7ea31a3c7586475514a7dacb0d9fa3"} Sep 05 07:49:22.290242 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:22.290195 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:22.295041 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:22.295019 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" Sep 05 07:49:22.305033 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:22.304986 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-hhvm4" podStartSLOduration=17.83562739 podStartE2EDuration="22.304970472s" totalImagesPullingTime="4.469343082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:22.304310733 +0000 UTC m=+64.913285859" watchObservedRunningTime="2026-09-05 07:49:22.304970472 +0000 UTC m=+64.913945639" Sep 05 07:49:22.847734 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:22.847699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:22.850173 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:22.850140 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a2bc7982-908a-4621-b8d8-962a902ff34d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-cf2bt\" (UID: \"a2bc7982-908a-4621-b8d8-962a902ff34d\") " pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:22.946616 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:22.946581 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gx2xl\"" Sep 05 07:49:22.955356 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:22.955330 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-cf2bt" Sep 05 07:49:23.242815 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.242731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-cf2bt"] Sep 05 07:49:23.246349 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:23.246324 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2bc7982_908a_4621_b8d8_962a902ff34d.slice/crio-d4085a39de4e920fda60674bb58ad925d6a64beef97d777f10279e89355d0065 WatchSource:0}: Error finding container d4085a39de4e920fda60674bb58ad925d6a64beef97d777f10279e89355d0065: Status 404 returned error can't find the container with id d4085a39de4e920fda60674bb58ad925d6a64beef97d777f10279e89355d0065 Sep 05 07:49:23.294410 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.294378 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cf2bt" event={"ID":"a2bc7982-908a-4621-b8d8-962a902ff34d","Type":"ContainerStarted","Data":"d4085a39de4e920fda60674bb58ad925d6a64beef97d777f10279e89355d0065"} Sep 05 07:49:23.754606 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.754564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:49:23.754927 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.754662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs\") pod \"network-metrics-daemon-zr6kd\" (UID: \"73836491-7624-4e24-a8cd-2ad60bc09155\") " pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:49:23.756906 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.756826 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 05 07:49:23.757325 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.757305 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 05 07:49:23.770236 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.768566 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/73836491-7624-4e24-a8cd-2ad60bc09155-metrics-certs\") pod \"network-metrics-daemon-zr6kd\" (UID: \"73836491-7624-4e24-a8cd-2ad60bc09155\") " pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:49:23.770236 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.768909 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b43f76a2-cd92-49a6-b567-71f3ad019d30-original-pull-secret\") pod \"global-pull-secret-syncer-xtq7n\" (UID: \"b43f76a2-cd92-49a6-b567-71f3ad019d30\") " pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:49:23.855819 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.855781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xt7z\" (UniqueName: \"kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z\") pod \"network-check-target-dvr7g\" (UID: \"dbfac494-fbe8-4979-ba88-fcf267911128\") " pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:49:23.858464 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.858443 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xt7z\" (UniqueName: \"kubernetes.io/projected/dbfac494-fbe8-4979-ba88-fcf267911128-kube-api-access-5xt7z\") pod \"network-check-target-dvr7g\" (UID: \"dbfac494-fbe8-4979-ba88-fcf267911128\") " pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:49:23.896246 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.896202 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hp5x9\"" Sep 05 07:49:23.903583 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.903553 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g9xp6\"" Sep 05 07:49:23.904605 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.904586 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zr6kd" Sep 05 07:49:23.906280 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.906257 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xtq7n" Sep 05 07:49:23.912351 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:23.912332 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:49:24.158161 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.158127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:49:24.158577 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.158191 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:49:24.160742 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.160723 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls\") pod \"image-registry-859f44b44c-q9ptx\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:49:24.160981 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.160964 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/332ec72e-6421-48a2-b0c0-def5d946f80c-metrics-tls\") pod \"dns-default-7cs9n\" (UID: \"332ec72e-6421-48a2-b0c0-def5d946f80c\") " pod="openshift-dns/dns-default-7cs9n" Sep 05 07:49:24.298320 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:24.295727 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43f76a2_cd92_49a6_b567_71f3ad019d30.slice/crio-41b4461cc640575db66360c3f9ab6620f3d2b41922cf0bc2f0a6e421819295c4 WatchSource:0}: Error finding container 41b4461cc640575db66360c3f9ab6620f3d2b41922cf0bc2f0a6e421819295c4: Status 404 returned error can't find the container with id 41b4461cc640575db66360c3f9ab6620f3d2b41922cf0bc2f0a6e421819295c4 Sep 05 07:49:24.306259 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.305746 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xtq7n"] Sep 05 07:49:24.307129 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.307094 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dvr7g"] Sep 05 07:49:24.307785 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.307709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zr6kd"] Sep 05 07:49:24.308385 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.308354 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cf2bt" event={"ID":"a2bc7982-908a-4621-b8d8-962a902ff34d","Type":"ContainerStarted","Data":"cc3012d01fc0eb9d88c0c4bf910abfcf5e443c91779af5d7caf786153d52b371"} Sep 05 07:49:24.308483 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.308418 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cf2bt" event={"ID":"a2bc7982-908a-4621-b8d8-962a902ff34d","Type":"ContainerStarted","Data":"2196f797eb7bf3d775dbcc608f7ab11b4a6ffd89f024058abf676b0dd5b9604c"} Sep 05 07:49:24.308890 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:24.308862 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73836491_7624_4e24_a8cd_2ad60bc09155.slice/crio-eeb9461f54fa3d3715666fe6aa29b4ec374653bbfffe17b795edd37b8cb115aa WatchSource:0}: Error finding container eeb9461f54fa3d3715666fe6aa29b4ec374653bbfffe17b795edd37b8cb115aa: Status 404 returned error can't find the container with id eeb9461f54fa3d3715666fe6aa29b4ec374653bbfffe17b795edd37b8cb115aa Sep 05 07:49:24.309686 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:24.309659 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfac494_fbe8_4979_ba88_fcf267911128.slice/crio-7f6530aea2e39840d63b6e58d8feeb0afac293143d3118f90db46573f603a942 WatchSource:0}: Error finding container 7f6530aea2e39840d63b6e58d8feeb0afac293143d3118f90db46573f603a942: Status 404 returned error can't find the container with id 7f6530aea2e39840d63b6e58d8feeb0afac293143d3118f90db46573f603a942 Sep 05 07:49:24.399057 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.399024 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-z2wx8\"" Sep 05 07:49:24.405272 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.405248 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t6wxv\"" Sep 05 07:49:24.407612 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.407597 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:49:24.414489 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.414469 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7cs9n" Sep 05 07:49:24.693154 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.693123 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-859f44b44c-q9ptx"] Sep 05 07:49:24.696409 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:24.696378 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1f7b6d9_cad6_43b9_ae4c_a283b9e4ecdc.slice/crio-7ccfc38185d307d70cf1bc3775bfa21b1ec011cb089444a85186029d3b257d60 WatchSource:0}: Error finding container 7ccfc38185d307d70cf1bc3775bfa21b1ec011cb089444a85186029d3b257d60: Status 404 returned error can't find the container with id 7ccfc38185d307d70cf1bc3775bfa21b1ec011cb089444a85186029d3b257d60 Sep 05 07:49:24.700432 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:24.700409 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7cs9n"] Sep 05 07:49:24.703864 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:24.703840 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332ec72e_6421_48a2_b0c0_def5d946f80c.slice/crio-959fdd153d938fe53c64d0e5b064addb1854b9bbce5bd415f50e3b1360395841 WatchSource:0}: Error finding container 959fdd153d938fe53c64d0e5b064addb1854b9bbce5bd415f50e3b1360395841: Status 404 returned error can't find the container with id 959fdd153d938fe53c64d0e5b064addb1854b9bbce5bd415f50e3b1360395841 Sep 05 07:49:25.316137 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:25.316098 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7cs9n" event={"ID":"332ec72e-6421-48a2-b0c0-def5d946f80c","Type":"ContainerStarted","Data":"959fdd153d938fe53c64d0e5b064addb1854b9bbce5bd415f50e3b1360395841"} Sep 05 07:49:25.318538 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:25.318505 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xtq7n" event={"ID":"b43f76a2-cd92-49a6-b567-71f3ad019d30","Type":"ContainerStarted","Data":"41b4461cc640575db66360c3f9ab6620f3d2b41922cf0bc2f0a6e421819295c4"} Sep 05 07:49:25.324574 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:25.324547 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" event={"ID":"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc","Type":"ContainerStarted","Data":"4e49620e38794eac22696f96a229563ba84893fa4246254d0ea18d22f4ac58d5"} Sep 05 07:49:25.324710 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:25.324591 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" event={"ID":"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc","Type":"ContainerStarted","Data":"7ccfc38185d307d70cf1bc3775bfa21b1ec011cb089444a85186029d3b257d60"} Sep 05 07:49:25.325288 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:25.325266 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:49:25.329344 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:25.329308 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dvr7g" event={"ID":"dbfac494-fbe8-4979-ba88-fcf267911128","Type":"ContainerStarted","Data":"32edcf69e17c4273c75abc0f109a101fafdb9708d3bca48de7346fdb9cfe7e7f"} Sep 05 07:49:25.329445 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:25.329359 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dvr7g" event={"ID":"dbfac494-fbe8-4979-ba88-fcf267911128","Type":"ContainerStarted","Data":"7f6530aea2e39840d63b6e58d8feeb0afac293143d3118f90db46573f603a942"} Sep 05 07:49:25.329512 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:25.329481 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:49:25.332304 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:25.332279 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zr6kd" event={"ID":"73836491-7624-4e24-a8cd-2ad60bc09155","Type":"ContainerStarted","Data":"eeb9461f54fa3d3715666fe6aa29b4ec374653bbfffe17b795edd37b8cb115aa"} Sep 05 07:49:25.345880 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:25.345537 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" podStartSLOduration=67.345516191 podStartE2EDuration="1m7.345516191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:25.34429263 +0000 UTC m=+67.953267764" watchObservedRunningTime="2026-09-05 07:49:25.345516191 +0000 UTC m=+67.954491315" Sep 05 07:49:25.360881 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:25.360687 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dvr7g" podStartSLOduration=67.360666245 podStartE2EDuration="1m7.360666245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:25.359376816 +0000 UTC m=+67.968351945" watchObservedRunningTime="2026-09-05 07:49:25.360666245 +0000 UTC m=+67.969641368" Sep 05 07:49:27.344243 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:27.343918 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zr6kd" event={"ID":"73836491-7624-4e24-a8cd-2ad60bc09155","Type":"ContainerStarted","Data":"e5fe0e79ae71aea5ed79fb88f02bdadd925825fcc1704f6f2e8831d5e0ba8c3d"} Sep 05 07:49:27.344243 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:27.343984 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zr6kd" event={"ID":"73836491-7624-4e24-a8cd-2ad60bc09155","Type":"ContainerStarted","Data":"c3ab0b797ba4ef7e000eb2f4c16aebe238d938b6a4925e596fbb4975efa24c94"} Sep 05 07:49:27.346059 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:27.346029 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7cs9n" event={"ID":"332ec72e-6421-48a2-b0c0-def5d946f80c","Type":"ContainerStarted","Data":"a0963fa3732e4be8a1f8db0c54377843ec602a94739d42cfac5f1ed7eaf7ba2c"} Sep 05 07:49:27.346190 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:27.346085 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7cs9n" event={"ID":"332ec72e-6421-48a2-b0c0-def5d946f80c","Type":"ContainerStarted","Data":"b8d4d206be800841f5e66ba18d01da8d0468f00c741816f45bae4fef2a0aba0a"} Sep 05 07:49:27.346190 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:27.346130 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7cs9n" Sep 05 07:49:27.348734 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:27.348701 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-cf2bt" event={"ID":"a2bc7982-908a-4621-b8d8-962a902ff34d","Type":"ContainerStarted","Data":"79f7c2d24c0c6b579d11e2b18583e7474a8dd5344ffa58219800ddd3292cd3e5"} Sep 05 07:49:27.360247 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:27.360183 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zr6kd" podStartSLOduration=66.984203177 podStartE2EDuration="1m9.360168325s" totalImagesPullingTime="2.375965148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:27.359101475 +0000 UTC m=+69.968076599" watchObservedRunningTime="2026-09-05 07:49:27.360168325 +0000 UTC m=+69.969143448" Sep 05 07:49:27.374970 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:27.374917 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7cs9n" podStartSLOduration=33.392675568 podStartE2EDuration="35.374897328s" totalImagesPullingTime="1.98222176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:48:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:27.373172846 +0000 UTC m=+69.982147970" watchObservedRunningTime="2026-09-05 07:49:27.374897328 +0000 UTC m=+69.983872509" Sep 05 07:49:27.388839 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:27.388787 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-cf2bt" podStartSLOduration=18.12481728 podStartE2EDuration="21.38876704s" totalImagesPullingTime="3.26394976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 07:49:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:27.388041145 +0000 UTC m=+69.997016297" watchObservedRunningTime="2026-09-05 07:49:27.38876704 +0000 UTC m=+69.997742166" Sep 05 07:49:29.357389 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:29.357352 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xtq7n" event={"ID":"b43f76a2-cd92-49a6-b567-71f3ad019d30","Type":"ContainerStarted","Data":"a36f3d292d0546716bc5caba91349a4a207b42a9ca95d5dd45f46dc68326590e"} Sep 05 07:49:29.371140 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:29.371082 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xtq7n" podStartSLOduration=67.120558614 podStartE2EDuration="1m11.37106288s" totalImagesPullingTime="4.250504266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 07:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:29.370624185 +0000 UTC m=+71.979599307" watchObservedRunningTime="2026-09-05 07:49:29.37106288 +0000 UTC m=+71.980038005" Sep 05 07:49:32.289573 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.289545 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p"] Sep 05 07:49:32.294480 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.294456 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.296706 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.296674 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 05 07:49:32.296706 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.296678 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 05 07:49:32.296913 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.296737 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 05 07:49:32.296913 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.296740 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 05 07:49:32.297178 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.297159 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 05 07:49:32.297686 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.297667 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 05 07:49:32.300457 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.300441 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 05 07:49:32.311575 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.311554 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p"] Sep 05 07:49:32.430131 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.430085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs5lx\" (UniqueName: \"kubernetes.io/projected/f740e967-a04f-426f-b084-77c19b63a0df-kube-api-access-qs5lx\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.430369 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.430143 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f740e967-a04f-426f-b084-77c19b63a0df-ca\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.430369 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.430172 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f740e967-a04f-426f-b084-77c19b63a0df-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.430369 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.430337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f740e967-a04f-426f-b084-77c19b63a0df-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.430528 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.430378 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f740e967-a04f-426f-b084-77c19b63a0df-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.430528 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.430427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f740e967-a04f-426f-b084-77c19b63a0df-hub\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.531824 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.531784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f740e967-a04f-426f-b084-77c19b63a0df-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.531824 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.531826 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f740e967-a04f-426f-b084-77c19b63a0df-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.532071 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.531863 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f740e967-a04f-426f-b084-77c19b63a0df-hub\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.532071 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.531886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs5lx\" (UniqueName: \"kubernetes.io/projected/f740e967-a04f-426f-b084-77c19b63a0df-kube-api-access-qs5lx\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.532071 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.531903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f740e967-a04f-426f-b084-77c19b63a0df-ca\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.532071 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.531917 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f740e967-a04f-426f-b084-77c19b63a0df-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.532808 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.532776 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/f740e967-a04f-426f-b084-77c19b63a0df-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.534673 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.534644 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/f740e967-a04f-426f-b084-77c19b63a0df-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.534774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.534711 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f740e967-a04f-426f-b084-77c19b63a0df-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.535009 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.534988 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/f740e967-a04f-426f-b084-77c19b63a0df-ca\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.535084 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.534997 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/f740e967-a04f-426f-b084-77c19b63a0df-hub\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.540495 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.540440 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs5lx\" (UniqueName: \"kubernetes.io/projected/f740e967-a04f-426f-b084-77c19b63a0df-kube-api-access-qs5lx\") pod \"cluster-proxy-proxy-agent-75fb4645df-csf6p\" (UID: \"f740e967-a04f-426f-b084-77c19b63a0df\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.614496 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.614464 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" Sep 05 07:49:32.897245 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:32.897197 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p"] Sep 05 07:49:32.900677 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:32.900646 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf740e967_a04f_426f_b084_77c19b63a0df.slice/crio-04fa7091d0a5a93c0112ed2d80b7a419385d067698f67d6257b5530a4bb32de7 WatchSource:0}: Error finding container 04fa7091d0a5a93c0112ed2d80b7a419385d067698f67d6257b5530a4bb32de7: Status 404 returned error can't find the container with id 04fa7091d0a5a93c0112ed2d80b7a419385d067698f67d6257b5530a4bb32de7 Sep 05 07:49:33.035506 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:33.035468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:33.038038 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:33.038008 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c0bd8583-070e-4b16-96bf-3780bbb5497f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-npnpg\" (UID: \"c0bd8583-070e-4b16-96bf-3780bbb5497f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:33.205097 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:33.205009 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xsf7v\"" Sep 05 07:49:33.213484 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:33.213461 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" Sep 05 07:49:33.379148 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:33.379112 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" event={"ID":"f740e967-a04f-426f-b084-77c19b63a0df","Type":"ContainerStarted","Data":"04fa7091d0a5a93c0112ed2d80b7a419385d067698f67d6257b5530a4bb32de7"} Sep 05 07:49:33.489967 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:33.489888 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg"] Sep 05 07:49:33.494552 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:33.494519 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bd8583_070e_4b16_96bf_3780bbb5497f.slice/crio-400f57a46c1cbaf23156dfea12cf144fe433c63793545256b930669b9aec7ffb WatchSource:0}: Error finding container 400f57a46c1cbaf23156dfea12cf144fe433c63793545256b930669b9aec7ffb: Status 404 returned error can't find the container with id 400f57a46c1cbaf23156dfea12cf144fe433c63793545256b930669b9aec7ffb Sep 05 07:49:34.387022 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:34.386980 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" event={"ID":"c0bd8583-070e-4b16-96bf-3780bbb5497f","Type":"ContainerStarted","Data":"400f57a46c1cbaf23156dfea12cf144fe433c63793545256b930669b9aec7ffb"} Sep 05 07:49:37.355269 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:37.355240 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7cs9n" Sep 05 07:49:37.403317 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:37.403246 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" event={"ID":"c0bd8583-070e-4b16-96bf-3780bbb5497f","Type":"ContainerStarted","Data":"d7db7328b6d77bb35e6072d78c82a22ec00b3e40c9b58ff8c838cd321a688d16"} Sep 05 07:49:37.405488 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:37.405461 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" event={"ID":"f740e967-a04f-426f-b084-77c19b63a0df","Type":"ContainerStarted","Data":"28cfb81e41ec6d976f729dd79f0a56a5c1cda2e9a24999b54d9beab6306e0ac4"} Sep 05 07:49:37.405607 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:37.405501 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" event={"ID":"f740e967-a04f-426f-b084-77c19b63a0df","Type":"ContainerStarted","Data":"2d33b41790e9d98dff9e13fbdd89fe5065730a890565862403d1b0f7cc03baf4"} Sep 05 07:49:37.430257 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:37.430175 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-npnpg" podStartSLOduration=32.770688122 podStartE2EDuration="36.430153563s" totalImagesPullingTime="3.659465441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:37.42948582 +0000 UTC m=+80.038460943" watchObservedRunningTime="2026-09-05 07:49:37.430153563 +0000 UTC m=+80.039128689" Sep 05 07:49:38.410983 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:38.410948 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" event={"ID":"f740e967-a04f-426f-b084-77c19b63a0df","Type":"ContainerStarted","Data":"0a5b782376362d37440ab313b723a1e296212db58c939797d50aa889c8995067"} Sep 05 07:49:38.431961 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:38.431908 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75fb4645df-csf6p" podStartSLOduration=2.16314554 podStartE2EDuration="6.431891423s" totalImagesPullingTime="4.268745883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:49:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:38.430287218 +0000 UTC m=+81.039262340" watchObservedRunningTime="2026-09-05 07:49:38.431891423 +0000 UTC m=+81.040866545" Sep 05 07:49:44.412513 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:44.412468 2576 patch_prober.go:28] interesting pod/image-registry-859f44b44c-q9ptx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 05 07:49:44.412988 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:44.412522 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" podUID="a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 05 07:49:47.316371 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.316331 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zx7wd"] Sep 05 07:49:47.322385 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.322358 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.324354 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.324326 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 05 07:49:47.324844 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.324552 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 05 07:49:47.324844 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.324573 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9tz6f\"" Sep 05 07:49:47.324844 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.324692 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 05 07:49:47.325192 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.325171 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 05 07:49:47.354871 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.354845 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:49:47.446403 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.446374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.446403 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.446407 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-tls\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.446669 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.446435 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrr6m\" (UniqueName: \"kubernetes.io/projected/58a109f7-4b95-4432-9289-ce6e72e852f6-kube-api-access-lrr6m\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.446669 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.446484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-wtmp\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.446669 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.446635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/58a109f7-4b95-4432-9289-ce6e72e852f6-root\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.446669 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.446660 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-textfile\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.446881 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.446689 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.446881 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.446716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58a109f7-4b95-4432-9289-ce6e72e852f6-sys\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.446881 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.446748 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58a109f7-4b95-4432-9289-ce6e72e852f6-metrics-client-ca\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.547941 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.547910 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548132 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.547948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-tls\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548132 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.547983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrr6m\" (UniqueName: \"kubernetes.io/projected/58a109f7-4b95-4432-9289-ce6e72e852f6-kube-api-access-lrr6m\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548132 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.548019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-wtmp\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548132 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.548072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/58a109f7-4b95-4432-9289-ce6e72e852f6-root\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548132 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.548124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/58a109f7-4b95-4432-9289-ce6e72e852f6-root\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548428 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.548161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-textfile\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548428 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.548190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548428 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.548244 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-wtmp\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548428 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.548256 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58a109f7-4b95-4432-9289-ce6e72e852f6-sys\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548428 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.548293 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58a109f7-4b95-4432-9289-ce6e72e852f6-sys\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548428 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.548323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58a109f7-4b95-4432-9289-ce6e72e852f6-metrics-client-ca\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548725 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.548510 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-textfile\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548725 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.548705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-accelerators-collector-config\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.548984 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.548869 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/58a109f7-4b95-4432-9289-ce6e72e852f6-metrics-client-ca\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.550597 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.550575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.550745 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.550723 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/58a109f7-4b95-4432-9289-ce6e72e852f6-node-exporter-tls\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.557439 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.557418 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrr6m\" (UniqueName: \"kubernetes.io/projected/58a109f7-4b95-4432-9289-ce6e72e852f6-kube-api-access-lrr6m\") pod \"node-exporter-zx7wd\" (UID: \"58a109f7-4b95-4432-9289-ce6e72e852f6\") " pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.633947 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:47.633856 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zx7wd" Sep 05 07:49:47.647038 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:49:47.646946 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a109f7_4b95_4432_9289_ce6e72e852f6.slice/crio-8b22f8926fa1d28cd08f36864113dd6579703d60285fa07e840ab8b56e5c39d4 WatchSource:0}: Error finding container 8b22f8926fa1d28cd08f36864113dd6579703d60285fa07e840ab8b56e5c39d4: Status 404 returned error can't find the container with id 8b22f8926fa1d28cd08f36864113dd6579703d60285fa07e840ab8b56e5c39d4 Sep 05 07:49:48.449205 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:48.449158 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zx7wd" event={"ID":"58a109f7-4b95-4432-9289-ce6e72e852f6","Type":"ContainerStarted","Data":"8b22f8926fa1d28cd08f36864113dd6579703d60285fa07e840ab8b56e5c39d4"} Sep 05 07:49:49.457287 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:49.457245 2576 generic.go:356] "Generic (PLEG): container finished" podID="58a109f7-4b95-4432-9289-ce6e72e852f6" containerID="6aa474535a6015f72f3f886a782514325f84913efcd06981557325526d89aaf1" exitCode=0 Sep 05 07:49:49.457687 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:49.457301 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zx7wd" event={"ID":"58a109f7-4b95-4432-9289-ce6e72e852f6","Type":"ContainerDied","Data":"6aa474535a6015f72f3f886a782514325f84913efcd06981557325526d89aaf1"} Sep 05 07:49:50.463646 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:50.463612 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zx7wd" event={"ID":"58a109f7-4b95-4432-9289-ce6e72e852f6","Type":"ContainerStarted","Data":"474145f1557d73dca53911f169e42a59ec47bfffd64ffd3cf36e8ee153e58eaa"} Sep 05 07:49:50.464135 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:50.463664 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zx7wd" event={"ID":"58a109f7-4b95-4432-9289-ce6e72e852f6","Type":"ContainerStarted","Data":"e4bce4790cc2164cf818d3d03b8ab439f2884ab28e65af09106c2e196eeda72e"} Sep 05 07:49:50.482034 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:50.481971 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zx7wd" podStartSLOduration=2.345924827 podStartE2EDuration="3.48195367s" totalImagesPullingTime="1.065761386s" totalInitContainerRuntime="70.267457ms" isStatefulPod=true podCreationTimestamp="2026-09-05 07:49:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:49:50.480772058 +0000 UTC m=+93.089747181" watchObservedRunningTime="2026-09-05 07:49:50.48195367 +0000 UTC m=+93.090928793" Sep 05 07:49:54.227830 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:54.227792 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-859f44b44c-q9ptx"] Sep 05 07:49:56.339162 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:49:56.339132 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dvr7g" Sep 05 07:50:19.248463 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.248415 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" podUID="a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" containerName="registry" containerID="cri-o://4e49620e38794eac22696f96a229563ba84893fa4246254d0ea18d22f4ac58d5" gracePeriod=30 Sep 05 07:50:19.489387 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.489361 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:50:19.568067 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.568036 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls\") pod \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " Sep 05 07:50:19.568268 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.568088 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-image-registry-private-configuration\") pod \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " Sep 05 07:50:19.568268 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.568108 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-bound-sa-token\") pod \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " Sep 05 07:50:19.568268 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.568142 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9lcs\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-kube-api-access-q9lcs\") pod \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " Sep 05 07:50:19.568268 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.568174 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-certificates\") pod \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " Sep 05 07:50:19.568268 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.568201 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-ca-trust-extracted\") pod \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " Sep 05 07:50:19.568268 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.568254 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-installation-pull-secrets\") pod \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " Sep 05 07:50:19.568580 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.568300 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-trusted-ca\") pod \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\" (UID: \"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc\") " Sep 05 07:50:19.568912 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.568869 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 07:50:19.568912 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.568878 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 07:50:19.570766 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.570739 2576 generic.go:356] "Generic (PLEG): container finished" podID="a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" containerID="4e49620e38794eac22696f96a229563ba84893fa4246254d0ea18d22f4ac58d5" exitCode=0 Sep 05 07:50:19.570892 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.570811 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" Sep 05 07:50:19.570892 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.570827 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" event={"ID":"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc","Type":"ContainerDied","Data":"4e49620e38794eac22696f96a229563ba84893fa4246254d0ea18d22f4ac58d5"} Sep 05 07:50:19.570892 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.570875 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859f44b44c-q9ptx" event={"ID":"a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc","Type":"ContainerDied","Data":"7ccfc38185d307d70cf1bc3775bfa21b1ec011cb089444a85186029d3b257d60"} Sep 05 07:50:19.571056 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.570900 2576 scope.go:117] "RemoveContainer" containerID="4e49620e38794eac22696f96a229563ba84893fa4246254d0ea18d22f4ac58d5" Sep 05 07:50:19.571200 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.571172 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-kube-api-access-q9lcs" (OuterVolumeSpecName: "kube-api-access-q9lcs") pod "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc"). InnerVolumeSpecName "kube-api-access-q9lcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:50:19.571538 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.571489 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 07:50:19.571651 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.571590 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:50:19.571716 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.571649 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:50:19.571822 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.571801 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 07:50:19.580745 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.580716 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" (UID: "a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:50:19.587862 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.587841 2576 scope.go:117] "RemoveContainer" containerID="4e49620e38794eac22696f96a229563ba84893fa4246254d0ea18d22f4ac58d5" Sep 05 07:50:19.588194 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:50:19.588166 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e49620e38794eac22696f96a229563ba84893fa4246254d0ea18d22f4ac58d5\": container with ID starting with 4e49620e38794eac22696f96a229563ba84893fa4246254d0ea18d22f4ac58d5 not found: ID does not exist" containerID="4e49620e38794eac22696f96a229563ba84893fa4246254d0ea18d22f4ac58d5" Sep 05 07:50:19.588295 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.588203 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e49620e38794eac22696f96a229563ba84893fa4246254d0ea18d22f4ac58d5"} err="failed to get container status \"4e49620e38794eac22696f96a229563ba84893fa4246254d0ea18d22f4ac58d5\": rpc error: code = NotFound desc = could not find container \"4e49620e38794eac22696f96a229563ba84893fa4246254d0ea18d22f4ac58d5\": container with ID starting with 4e49620e38794eac22696f96a229563ba84893fa4246254d0ea18d22f4ac58d5 not found: ID does not exist" Sep 05 07:50:19.669717 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.669684 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-trusted-ca\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:50:19.669717 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.669712 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-tls\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:50:19.669717 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.669722 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-image-registry-private-configuration\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:50:19.669952 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.669732 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-bound-sa-token\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:50:19.669952 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.669741 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9lcs\" (UniqueName: \"kubernetes.io/projected/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-kube-api-access-q9lcs\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:50:19.669952 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.669749 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-registry-certificates\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:50:19.669952 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.669756 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-ca-trust-extracted\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:50:19.669952 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.669765 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc-installation-pull-secrets\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:50:19.890594 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.890560 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-859f44b44c-q9ptx"] Sep 05 07:50:19.894767 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:19.894744 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-859f44b44c-q9ptx"] Sep 05 07:50:21.885487 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:21.885457 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" path="/var/lib/kubelet/pods/a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc/volumes" Sep 05 07:50:32.627074 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:32.627035 2576 generic.go:356] "Generic (PLEG): container finished" podID="e0902955-102e-4493-83f5-a427d9f7c1e8" containerID="a9773b22e45baac0cec37a7da48bd03356729b7b250310c8101b299dcfe932da" exitCode=0 Sep 05 07:50:32.627522 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:32.627097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" event={"ID":"e0902955-102e-4493-83f5-a427d9f7c1e8","Type":"ContainerDied","Data":"a9773b22e45baac0cec37a7da48bd03356729b7b250310c8101b299dcfe932da"} Sep 05 07:50:32.627522 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:32.627399 2576 scope.go:117] "RemoveContainer" containerID="a9773b22e45baac0cec37a7da48bd03356729b7b250310c8101b299dcfe932da" Sep 05 07:50:33.634779 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:50:33.634737 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-dp8m7" event={"ID":"e0902955-102e-4493-83f5-a427d9f7c1e8","Type":"ContainerStarted","Data":"88db08ea541e33899cc9620b703e989c793f663d1ebb5bf07706a28bfcc32e67"} Sep 05 07:52:13.331142 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.331108 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk"] Sep 05 07:52:13.331625 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.331387 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" containerName="registry" Sep 05 07:52:13.331625 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.331398 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" containerName="registry" Sep 05 07:52:13.331625 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.331440 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1f7b6d9-cad6-43b9-ae4c-a283b9e4ecdc" containerName="registry" Sep 05 07:52:13.334181 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.334165 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" Sep 05 07:52:13.336176 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.336158 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 05 07:52:13.336303 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.336249 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 05 07:52:13.336303 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.336279 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cns7m\"" Sep 05 07:52:13.343652 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.343629 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk"] Sep 05 07:52:13.412961 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.412927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk\" (UID: \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" Sep 05 07:52:13.413159 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.412976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfh2\" (UniqueName: \"kubernetes.io/projected/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-kube-api-access-brfh2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk\" (UID: \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" Sep 05 07:52:13.413159 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.413049 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk\" (UID: \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" Sep 05 07:52:13.513892 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.513852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk\" (UID: \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" Sep 05 07:52:13.514096 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.513914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfh2\" (UniqueName: \"kubernetes.io/projected/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-kube-api-access-brfh2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk\" (UID: \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" Sep 05 07:52:13.514096 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.513966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk\" (UID: \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" Sep 05 07:52:13.514356 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.514333 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk\" (UID: \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" Sep 05 07:52:13.514439 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.514344 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk\" (UID: \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" Sep 05 07:52:13.522249 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.522198 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfh2\" (UniqueName: \"kubernetes.io/projected/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-kube-api-access-brfh2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk\" (UID: \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" Sep 05 07:52:13.643406 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.643310 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" Sep 05 07:52:13.915981 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.915900 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk"] Sep 05 07:52:13.919020 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:52:13.918985 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a95bc7_e4af_4c0d_91e7_aeebd3e7cf1e.slice/crio-99ebe60491a1a05e9fe1b8f57f23786586cec2f067829b7c61233201392466f3 WatchSource:0}: Error finding container 99ebe60491a1a05e9fe1b8f57f23786586cec2f067829b7c61233201392466f3: Status 404 returned error can't find the container with id 99ebe60491a1a05e9fe1b8f57f23786586cec2f067829b7c61233201392466f3 Sep 05 07:52:13.982233 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:13.982183 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" event={"ID":"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e","Type":"ContainerStarted","Data":"99ebe60491a1a05e9fe1b8f57f23786586cec2f067829b7c61233201392466f3"} Sep 05 07:52:20.007139 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:20.007110 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" event={"ID":"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e","Type":"ContainerStarted","Data":"ada04af4f802c7ca2f84746964dfe7863ec1ca8749629c4f1483ff7568cf3889"} Sep 05 07:52:21.012409 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:21.012368 2576 generic.go:356] "Generic (PLEG): container finished" podID="d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e" containerID="ada04af4f802c7ca2f84746964dfe7863ec1ca8749629c4f1483ff7568cf3889" exitCode=0 Sep 05 07:52:21.013254 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:21.012453 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" event={"ID":"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e","Type":"ContainerDied","Data":"ada04af4f802c7ca2f84746964dfe7863ec1ca8749629c4f1483ff7568cf3889"} Sep 05 07:52:24.024663 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:24.024627 2576 generic.go:356] "Generic (PLEG): container finished" podID="d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e" containerID="e28f0f90b1127b04d202f2532a62bf117003539b212c1bf02bfe3eb689674963" exitCode=0 Sep 05 07:52:24.025068 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:24.024680 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" event={"ID":"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e","Type":"ContainerDied","Data":"e28f0f90b1127b04d202f2532a62bf117003539b212c1bf02bfe3eb689674963"} Sep 05 07:52:31.053937 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:31.053896 2576 generic.go:356] "Generic (PLEG): container finished" podID="d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e" containerID="c046dff2d43cafe16fc0dec5d6433e4c10a1334ccf2c4882abf8ccf963a0532d" exitCode=0 Sep 05 07:52:31.054364 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:31.053962 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" event={"ID":"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e","Type":"ContainerDied","Data":"c046dff2d43cafe16fc0dec5d6433e4c10a1334ccf2c4882abf8ccf963a0532d"} Sep 05 07:52:32.179647 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:32.179619 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" Sep 05 07:52:32.260968 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:32.260933 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-bundle\") pod \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\" (UID: \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\") " Sep 05 07:52:32.260968 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:32.260971 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-util\") pod \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\" (UID: \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\") " Sep 05 07:52:32.261202 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:32.261001 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brfh2\" (UniqueName: \"kubernetes.io/projected/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-kube-api-access-brfh2\") pod \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\" (UID: \"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e\") " Sep 05 07:52:32.261639 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:32.261615 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-bundle" (OuterVolumeSpecName: "bundle") pod "d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e" (UID: "d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:52:32.263338 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:32.263278 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-kube-api-access-brfh2" (OuterVolumeSpecName: "kube-api-access-brfh2") pod "d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e" (UID: "d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e"). InnerVolumeSpecName "kube-api-access-brfh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:52:32.265361 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:32.265334 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-util" (OuterVolumeSpecName: "util") pod "d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e" (UID: "d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:52:32.361457 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:32.361426 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brfh2\" (UniqueName: \"kubernetes.io/projected/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-kube-api-access-brfh2\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:52:32.361457 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:32.361454 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-bundle\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:52:32.361633 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:32.361467 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e-util\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:52:33.062673 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:33.062649 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" Sep 05 07:52:33.062833 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:33.062644 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540z2bpk" event={"ID":"d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e","Type":"ContainerDied","Data":"99ebe60491a1a05e9fe1b8f57f23786586cec2f067829b7c61233201392466f3"} Sep 05 07:52:33.062833 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:33.062764 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99ebe60491a1a05e9fe1b8f57f23786586cec2f067829b7c61233201392466f3" Sep 05 07:52:40.829056 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.828963 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8"] Sep 05 07:52:40.829532 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.829350 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e" containerName="extract" Sep 05 07:52:40.829532 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.829368 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e" containerName="extract" Sep 05 07:52:40.829532 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.829385 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e" containerName="pull" Sep 05 07:52:40.829532 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.829394 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e" containerName="pull" Sep 05 07:52:40.829532 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.829414 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e" containerName="util" Sep 05 07:52:40.829532 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.829422 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e" containerName="util" Sep 05 07:52:40.829532 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.829483 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5a95bc7-e4af-4c0d-91e7-aeebd3e7cf1e" containerName="extract" Sep 05 07:52:40.831431 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.831412 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" Sep 05 07:52:40.833918 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.833898 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 05 07:52:40.834437 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.834415 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cns7m\"" Sep 05 07:52:40.834437 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.834424 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 05 07:52:40.842013 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.841993 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8"] Sep 05 07:52:40.921689 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.921654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af949c4-6206-48fb-961b-66af5ab0763a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8\" (UID: \"9af949c4-6206-48fb-961b-66af5ab0763a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" Sep 05 07:52:40.921689 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.921691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9af949c4-6206-48fb-961b-66af5ab0763a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8\" (UID: \"9af949c4-6206-48fb-961b-66af5ab0763a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" Sep 05 07:52:40.921894 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:40.921757 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs58d\" (UniqueName: \"kubernetes.io/projected/9af949c4-6206-48fb-961b-66af5ab0763a-kube-api-access-rs58d\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8\" (UID: \"9af949c4-6206-48fb-961b-66af5ab0763a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" Sep 05 07:52:41.022111 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:41.022073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af949c4-6206-48fb-961b-66af5ab0763a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8\" (UID: \"9af949c4-6206-48fb-961b-66af5ab0763a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" Sep 05 07:52:41.022111 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:41.022111 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9af949c4-6206-48fb-961b-66af5ab0763a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8\" (UID: \"9af949c4-6206-48fb-961b-66af5ab0763a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" Sep 05 07:52:41.022335 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:41.022170 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs58d\" (UniqueName: \"kubernetes.io/projected/9af949c4-6206-48fb-961b-66af5ab0763a-kube-api-access-rs58d\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8\" (UID: \"9af949c4-6206-48fb-961b-66af5ab0763a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" Sep 05 07:52:41.022571 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:41.022549 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9af949c4-6206-48fb-961b-66af5ab0763a-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8\" (UID: \"9af949c4-6206-48fb-961b-66af5ab0763a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" Sep 05 07:52:41.022608 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:41.022569 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af949c4-6206-48fb-961b-66af5ab0763a-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8\" (UID: \"9af949c4-6206-48fb-961b-66af5ab0763a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" Sep 05 07:52:41.030981 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:41.030957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs58d\" (UniqueName: \"kubernetes.io/projected/9af949c4-6206-48fb-961b-66af5ab0763a-kube-api-access-rs58d\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8\" (UID: \"9af949c4-6206-48fb-961b-66af5ab0763a\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" Sep 05 07:52:41.139722 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:41.139641 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" Sep 05 07:52:41.412601 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:41.412526 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8"] Sep 05 07:52:41.415334 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:52:41.415296 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af949c4_6206_48fb_961b_66af5ab0763a.slice/crio-877f575c58f2b55b0733de755566453a9b90a962293e5036a179b83c2015642b WatchSource:0}: Error finding container 877f575c58f2b55b0733de755566453a9b90a962293e5036a179b83c2015642b: Status 404 returned error can't find the container with id 877f575c58f2b55b0733de755566453a9b90a962293e5036a179b83c2015642b Sep 05 07:52:42.095507 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:42.095472 2576 generic.go:356] "Generic (PLEG): container finished" podID="9af949c4-6206-48fb-961b-66af5ab0763a" containerID="5eafb05a72fd16cb4f7184db7e5a7a9cac94170d0be0d7a1a9de464b16cad86d" exitCode=0 Sep 05 07:52:42.095878 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:42.095558 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" event={"ID":"9af949c4-6206-48fb-961b-66af5ab0763a","Type":"ContainerDied","Data":"5eafb05a72fd16cb4f7184db7e5a7a9cac94170d0be0d7a1a9de464b16cad86d"} Sep 05 07:52:42.095878 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:42.095602 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" event={"ID":"9af949c4-6206-48fb-961b-66af5ab0763a","Type":"ContainerStarted","Data":"877f575c58f2b55b0733de755566453a9b90a962293e5036a179b83c2015642b"} Sep 05 07:52:45.109154 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:45.109120 2576 generic.go:356] "Generic (PLEG): container finished" podID="9af949c4-6206-48fb-961b-66af5ab0763a" containerID="099ddb85ad1fd5052065b43997af78d5256da511ad5d602dcd09a8c6c39248a4" exitCode=0 Sep 05 07:52:45.109579 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:45.109163 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" event={"ID":"9af949c4-6206-48fb-961b-66af5ab0763a","Type":"ContainerDied","Data":"099ddb85ad1fd5052065b43997af78d5256da511ad5d602dcd09a8c6c39248a4"} Sep 05 07:52:46.115316 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:46.115265 2576 generic.go:356] "Generic (PLEG): container finished" podID="9af949c4-6206-48fb-961b-66af5ab0763a" containerID="d29ab73532fd8014cd28ebb4f8ff782cf3ead7233d32ab1e95954fcb255f6630" exitCode=0 Sep 05 07:52:46.115703 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:46.115332 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" event={"ID":"9af949c4-6206-48fb-961b-66af5ab0763a","Type":"ContainerDied","Data":"d29ab73532fd8014cd28ebb4f8ff782cf3ead7233d32ab1e95954fcb255f6630"} Sep 05 07:52:47.242916 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:47.242891 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" Sep 05 07:52:47.273540 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:47.273512 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9af949c4-6206-48fb-961b-66af5ab0763a-util\") pod \"9af949c4-6206-48fb-961b-66af5ab0763a\" (UID: \"9af949c4-6206-48fb-961b-66af5ab0763a\") " Sep 05 07:52:47.273691 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:47.273574 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af949c4-6206-48fb-961b-66af5ab0763a-bundle\") pod \"9af949c4-6206-48fb-961b-66af5ab0763a\" (UID: \"9af949c4-6206-48fb-961b-66af5ab0763a\") " Sep 05 07:52:47.273691 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:47.273620 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs58d\" (UniqueName: \"kubernetes.io/projected/9af949c4-6206-48fb-961b-66af5ab0763a-kube-api-access-rs58d\") pod \"9af949c4-6206-48fb-961b-66af5ab0763a\" (UID: \"9af949c4-6206-48fb-961b-66af5ab0763a\") " Sep 05 07:52:47.273981 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:47.273959 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af949c4-6206-48fb-961b-66af5ab0763a-bundle" (OuterVolumeSpecName: "bundle") pod "9af949c4-6206-48fb-961b-66af5ab0763a" (UID: "9af949c4-6206-48fb-961b-66af5ab0763a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:52:47.275910 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:47.275879 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9af949c4-6206-48fb-961b-66af5ab0763a-kube-api-access-rs58d" (OuterVolumeSpecName: "kube-api-access-rs58d") pod "9af949c4-6206-48fb-961b-66af5ab0763a" (UID: "9af949c4-6206-48fb-961b-66af5ab0763a"). InnerVolumeSpecName "kube-api-access-rs58d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:52:47.278876 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:47.278852 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9af949c4-6206-48fb-961b-66af5ab0763a-util" (OuterVolumeSpecName: "util") pod "9af949c4-6206-48fb-961b-66af5ab0763a" (UID: "9af949c4-6206-48fb-961b-66af5ab0763a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:52:47.374488 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:47.374394 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs58d\" (UniqueName: \"kubernetes.io/projected/9af949c4-6206-48fb-961b-66af5ab0763a-kube-api-access-rs58d\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:52:47.374488 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:47.374436 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9af949c4-6206-48fb-961b-66af5ab0763a-util\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:52:47.374488 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:47.374445 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9af949c4-6206-48fb-961b-66af5ab0763a-bundle\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:52:48.124700 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.124670 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" Sep 05 07:52:48.124700 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.124691 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87flfqj8" event={"ID":"9af949c4-6206-48fb-961b-66af5ab0763a","Type":"ContainerDied","Data":"877f575c58f2b55b0733de755566453a9b90a962293e5036a179b83c2015642b"} Sep 05 07:52:48.124908 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.124734 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="877f575c58f2b55b0733de755566453a9b90a962293e5036a179b83c2015642b" Sep 05 07:52:48.151532 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.151446 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8xrq8"] Sep 05 07:52:48.151712 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.151698 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af949c4-6206-48fb-961b-66af5ab0763a" containerName="pull" Sep 05 07:52:48.151749 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.151713 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af949c4-6206-48fb-961b-66af5ab0763a" containerName="pull" Sep 05 07:52:48.151749 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.151725 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af949c4-6206-48fb-961b-66af5ab0763a" containerName="util" Sep 05 07:52:48.151749 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.151730 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af949c4-6206-48fb-961b-66af5ab0763a" containerName="util" Sep 05 07:52:48.151749 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.151738 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9af949c4-6206-48fb-961b-66af5ab0763a" containerName="extract" Sep 05 07:52:48.151749 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.151743 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9af949c4-6206-48fb-961b-66af5ab0763a" containerName="extract" Sep 05 07:52:48.151888 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.151787 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9af949c4-6206-48fb-961b-66af5ab0763a" containerName="extract" Sep 05 07:52:48.153378 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.153363 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8xrq8" Sep 05 07:52:48.155293 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.155275 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 05 07:52:48.155386 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.155334 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 05 07:52:48.155436 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.155406 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-6xkz9\"" Sep 05 07:52:48.165232 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.162617 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8xrq8"] Sep 05 07:52:48.179396 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.179369 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/553faf8a-b095-4267-b546-7f46f3a0c86c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8xrq8\" (UID: \"553faf8a-b095-4267-b546-7f46f3a0c86c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xrq8" Sep 05 07:52:48.179509 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.179407 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxqhz\" (UniqueName: \"kubernetes.io/projected/553faf8a-b095-4267-b546-7f46f3a0c86c-kube-api-access-vxqhz\") pod \"cert-manager-webhook-6486ffd997-8xrq8\" (UID: \"553faf8a-b095-4267-b546-7f46f3a0c86c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xrq8" Sep 05 07:52:48.279746 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.279710 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/553faf8a-b095-4267-b546-7f46f3a0c86c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8xrq8\" (UID: \"553faf8a-b095-4267-b546-7f46f3a0c86c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xrq8" Sep 05 07:52:48.279746 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.279752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxqhz\" (UniqueName: \"kubernetes.io/projected/553faf8a-b095-4267-b546-7f46f3a0c86c-kube-api-access-vxqhz\") pod \"cert-manager-webhook-6486ffd997-8xrq8\" (UID: \"553faf8a-b095-4267-b546-7f46f3a0c86c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xrq8" Sep 05 07:52:48.290326 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.290291 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/553faf8a-b095-4267-b546-7f46f3a0c86c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-8xrq8\" (UID: \"553faf8a-b095-4267-b546-7f46f3a0c86c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xrq8" Sep 05 07:52:48.290447 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.290422 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxqhz\" (UniqueName: \"kubernetes.io/projected/553faf8a-b095-4267-b546-7f46f3a0c86c-kube-api-access-vxqhz\") pod \"cert-manager-webhook-6486ffd997-8xrq8\" (UID: \"553faf8a-b095-4267-b546-7f46f3a0c86c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-8xrq8" Sep 05 07:52:48.463241 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.463120 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-8xrq8" Sep 05 07:52:48.733083 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:48.733000 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-8xrq8"] Sep 05 07:52:48.736307 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:52:48.736267 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod553faf8a_b095_4267_b546_7f46f3a0c86c.slice/crio-fd41c9cbac27c80ef418f503685570d2078b2185d5e5c169eaa6404d3a26ec4a WatchSource:0}: Error finding container fd41c9cbac27c80ef418f503685570d2078b2185d5e5c169eaa6404d3a26ec4a: Status 404 returned error can't find the container with id fd41c9cbac27c80ef418f503685570d2078b2185d5e5c169eaa6404d3a26ec4a Sep 05 07:52:49.129596 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:49.129557 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8xrq8" event={"ID":"553faf8a-b095-4267-b546-7f46f3a0c86c","Type":"ContainerStarted","Data":"fd41c9cbac27c80ef418f503685570d2078b2185d5e5c169eaa6404d3a26ec4a"} Sep 05 07:52:52.142458 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:52.142423 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-8xrq8" event={"ID":"553faf8a-b095-4267-b546-7f46f3a0c86c","Type":"ContainerStarted","Data":"6b640bd164ce69fdb01410d2b5885a7309930cb614647e3b568a5709dd1e1477"} Sep 05 07:52:52.142848 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:52.142478 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8xrq8" Sep 05 07:52:52.180998 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:52.180947 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-8xrq8" podStartSLOduration=1.748093841 podStartE2EDuration="4.180929914s" totalImagesPullingTime="2.432836073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:52:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:52:52.180084246 +0000 UTC m=+274.789059370" watchObservedRunningTime="2026-09-05 07:52:52.180929914 +0000 UTC m=+274.789905036" Sep 05 07:52:58.148501 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:52:58.148468 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-8xrq8" Sep 05 07:53:00.380595 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.380552 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk"] Sep 05 07:53:00.386766 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.386746 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" Sep 05 07:53:00.388886 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.388853 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 05 07:53:00.389018 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.389004 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 05 07:53:00.389937 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.389753 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cns7m\"" Sep 05 07:53:00.391688 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.391666 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk"] Sep 05 07:53:00.463707 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.463676 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk\" (UID: \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" Sep 05 07:53:00.463707 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.463710 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk\" (UID: \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" Sep 05 07:53:00.463921 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.463782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwjzp\" (UniqueName: \"kubernetes.io/projected/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-kube-api-access-cwjzp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk\" (UID: \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" Sep 05 07:53:00.564686 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.564646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwjzp\" (UniqueName: \"kubernetes.io/projected/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-kube-api-access-cwjzp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk\" (UID: \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" Sep 05 07:53:00.564686 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.564690 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk\" (UID: \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" Sep 05 07:53:00.564917 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.564803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk\" (UID: \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" Sep 05 07:53:00.565008 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.564994 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk\" (UID: \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" Sep 05 07:53:00.565052 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.565040 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk\" (UID: \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" Sep 05 07:53:00.572816 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.572783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwjzp\" (UniqueName: \"kubernetes.io/projected/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-kube-api-access-cwjzp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk\" (UID: \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" Sep 05 07:53:00.697930 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.697838 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" Sep 05 07:53:00.968041 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:00.967949 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk"] Sep 05 07:53:00.972324 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:53:00.972295 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b20ee2_a8b7_4709_a5a1_e6bc83e73916.slice/crio-9997f346f260aba7456533ce0aff3a55f3c5a21216c642d1560d1df6d54ad1a4 WatchSource:0}: Error finding container 9997f346f260aba7456533ce0aff3a55f3c5a21216c642d1560d1df6d54ad1a4: Status 404 returned error can't find the container with id 9997f346f260aba7456533ce0aff3a55f3c5a21216c642d1560d1df6d54ad1a4 Sep 05 07:53:01.175797 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:01.175752 2576 generic.go:356] "Generic (PLEG): container finished" podID="a9b20ee2-a8b7-4709-a5a1-e6bc83e73916" containerID="f131bd5fb78c37931bbbd425526ff922fc86bc4cc0cb928a73076eb95f0f0545" exitCode=0 Sep 05 07:53:01.175981 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:01.175805 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" event={"ID":"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916","Type":"ContainerDied","Data":"f131bd5fb78c37931bbbd425526ff922fc86bc4cc0cb928a73076eb95f0f0545"} Sep 05 07:53:01.175981 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:01.175880 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" event={"ID":"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916","Type":"ContainerStarted","Data":"9997f346f260aba7456533ce0aff3a55f3c5a21216c642d1560d1df6d54ad1a4"} Sep 05 07:53:02.181628 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:02.181524 2576 generic.go:356] "Generic (PLEG): container finished" podID="a9b20ee2-a8b7-4709-a5a1-e6bc83e73916" containerID="1dbd9a52a1d10c5f4fade59048063b73113cc190e07839b559df5062dbc24347" exitCode=0 Sep 05 07:53:02.181628 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:02.181587 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" event={"ID":"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916","Type":"ContainerDied","Data":"1dbd9a52a1d10c5f4fade59048063b73113cc190e07839b559df5062dbc24347"} Sep 05 07:53:03.187097 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:03.187055 2576 generic.go:356] "Generic (PLEG): container finished" podID="a9b20ee2-a8b7-4709-a5a1-e6bc83e73916" containerID="445372d4ea8fc2c870461bcc2b3dd3e8faf8e6609d4ff7686788d2babcf0c059" exitCode=0 Sep 05 07:53:03.187517 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:03.187129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" event={"ID":"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916","Type":"ContainerDied","Data":"445372d4ea8fc2c870461bcc2b3dd3e8faf8e6609d4ff7686788d2babcf0c059"} Sep 05 07:53:04.312940 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:04.312915 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" Sep 05 07:53:04.390996 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:04.390964 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-util\") pod \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\" (UID: \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\") " Sep 05 07:53:04.391137 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:04.391027 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwjzp\" (UniqueName: \"kubernetes.io/projected/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-kube-api-access-cwjzp\") pod \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\" (UID: \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\") " Sep 05 07:53:04.391137 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:04.391068 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-bundle\") pod \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\" (UID: \"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916\") " Sep 05 07:53:04.391801 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:04.391775 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-bundle" (OuterVolumeSpecName: "bundle") pod "a9b20ee2-a8b7-4709-a5a1-e6bc83e73916" (UID: "a9b20ee2-a8b7-4709-a5a1-e6bc83e73916"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:53:04.393289 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:04.393263 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-kube-api-access-cwjzp" (OuterVolumeSpecName: "kube-api-access-cwjzp") pod "a9b20ee2-a8b7-4709-a5a1-e6bc83e73916" (UID: "a9b20ee2-a8b7-4709-a5a1-e6bc83e73916"). InnerVolumeSpecName "kube-api-access-cwjzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:53:04.399880 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:04.399841 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-util" (OuterVolumeSpecName: "util") pod "a9b20ee2-a8b7-4709-a5a1-e6bc83e73916" (UID: "a9b20ee2-a8b7-4709-a5a1-e6bc83e73916"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:53:04.492630 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:04.492537 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-bundle\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:53:04.492630 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:04.492573 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-util\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:53:04.492630 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:04.492582 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwjzp\" (UniqueName: \"kubernetes.io/projected/a9b20ee2-a8b7-4709-a5a1-e6bc83e73916-kube-api-access-cwjzp\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:53:05.197142 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:05.197101 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" event={"ID":"a9b20ee2-a8b7-4709-a5a1-e6bc83e73916","Type":"ContainerDied","Data":"9997f346f260aba7456533ce0aff3a55f3c5a21216c642d1560d1df6d54ad1a4"} Sep 05 07:53:05.197142 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:05.197121 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afcn8zk" Sep 05 07:53:05.197373 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:05.197155 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9997f346f260aba7456533ce0aff3a55f3c5a21216c642d1560d1df6d54ad1a4" Sep 05 07:53:17.624847 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.624813 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx"] Sep 05 07:53:17.625299 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.625056 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b20ee2-a8b7-4709-a5a1-e6bc83e73916" containerName="pull" Sep 05 07:53:17.625299 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.625067 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b20ee2-a8b7-4709-a5a1-e6bc83e73916" containerName="pull" Sep 05 07:53:17.625299 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.625085 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b20ee2-a8b7-4709-a5a1-e6bc83e73916" containerName="util" Sep 05 07:53:17.625299 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.625090 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b20ee2-a8b7-4709-a5a1-e6bc83e73916" containerName="util" Sep 05 07:53:17.625299 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.625096 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b20ee2-a8b7-4709-a5a1-e6bc83e73916" containerName="extract" Sep 05 07:53:17.625299 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.625102 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b20ee2-a8b7-4709-a5a1-e6bc83e73916" containerName="extract" Sep 05 07:53:17.625299 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.625151 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b20ee2-a8b7-4709-a5a1-e6bc83e73916" containerName="extract" Sep 05 07:53:17.626956 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.626941 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" Sep 05 07:53:17.629920 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.629901 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 05 07:53:17.630525 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.630510 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 05 07:53:17.634860 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.634842 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cns7m\"" Sep 05 07:53:17.653425 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.653403 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx"] Sep 05 07:53:17.689679 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.689642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfxt6\" (UniqueName: \"kubernetes.io/projected/dabdb768-8330-431b-ab10-a54f863c7500-kube-api-access-xfxt6\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx\" (UID: \"dabdb768-8330-431b-ab10-a54f863c7500\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" Sep 05 07:53:17.689859 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.689702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dabdb768-8330-431b-ab10-a54f863c7500-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx\" (UID: \"dabdb768-8330-431b-ab10-a54f863c7500\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" Sep 05 07:53:17.689859 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.689769 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dabdb768-8330-431b-ab10-a54f863c7500-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx\" (UID: \"dabdb768-8330-431b-ab10-a54f863c7500\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" Sep 05 07:53:17.790759 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.790720 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfxt6\" (UniqueName: \"kubernetes.io/projected/dabdb768-8330-431b-ab10-a54f863c7500-kube-api-access-xfxt6\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx\" (UID: \"dabdb768-8330-431b-ab10-a54f863c7500\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" Sep 05 07:53:17.790957 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.790775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dabdb768-8330-431b-ab10-a54f863c7500-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx\" (UID: \"dabdb768-8330-431b-ab10-a54f863c7500\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" Sep 05 07:53:17.790957 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.790813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dabdb768-8330-431b-ab10-a54f863c7500-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx\" (UID: \"dabdb768-8330-431b-ab10-a54f863c7500\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" Sep 05 07:53:17.791132 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.791116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dabdb768-8330-431b-ab10-a54f863c7500-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx\" (UID: \"dabdb768-8330-431b-ab10-a54f863c7500\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" Sep 05 07:53:17.791269 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.791196 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dabdb768-8330-431b-ab10-a54f863c7500-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx\" (UID: \"dabdb768-8330-431b-ab10-a54f863c7500\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" Sep 05 07:53:17.803925 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.803894 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 05 07:53:17.804277 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.804256 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l"] Sep 05 07:53:17.807611 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.807592 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.809710 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.809692 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 05 07:53:17.810268 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.810250 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 05 07:53:17.810800 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.810655 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 05 07:53:17.810800 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.810788 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 05 07:53:17.811033 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.810993 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mb5bz\"" Sep 05 07:53:17.812225 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.812189 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 05 07:53:17.813603 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.813586 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 07:53:17.818055 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.818035 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 07:53:17.818417 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.818395 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 07:53:17.823305 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.823283 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfxt6\" (UniqueName: \"kubernetes.io/projected/dabdb768-8330-431b-ab10-a54f863c7500-kube-api-access-xfxt6\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx\" (UID: \"dabdb768-8330-431b-ab10-a54f863c7500\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" Sep 05 07:53:17.824279 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.824196 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 07:53:17.827883 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.827865 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l"] Sep 05 07:53:17.829769 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.829729 2576 kubelet.go:1648] "Image garbage collection succeeded" Sep 05 07:53:17.891295 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.891105 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b69nr\" (UniqueName: \"kubernetes.io/projected/a545c090-555e-45cd-ac7d-6e9e3625aaf6-kube-api-access-b69nr\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.891295 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.891138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a545c090-555e-45cd-ac7d-6e9e3625aaf6-manifests\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.891295 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.891158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a545c090-555e-45cd-ac7d-6e9e3625aaf6-tmp\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.891295 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.891260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a545c090-555e-45cd-ac7d-6e9e3625aaf6-webhook-cert\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.895965 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.891300 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a545c090-555e-45cd-ac7d-6e9e3625aaf6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.938460 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.938430 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cns7m\"" Sep 05 07:53:17.946165 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.946143 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" Sep 05 07:53:17.992628 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.992587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a545c090-555e-45cd-ac7d-6e9e3625aaf6-webhook-cert\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.992812 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.992651 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a545c090-555e-45cd-ac7d-6e9e3625aaf6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.992812 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.992701 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b69nr\" (UniqueName: \"kubernetes.io/projected/a545c090-555e-45cd-ac7d-6e9e3625aaf6-kube-api-access-b69nr\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.992812 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.992746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a545c090-555e-45cd-ac7d-6e9e3625aaf6-manifests\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.992812 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.992778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a545c090-555e-45cd-ac7d-6e9e3625aaf6-tmp\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.993979 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.993907 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a545c090-555e-45cd-ac7d-6e9e3625aaf6-manifests\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.995991 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.995965 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a545c090-555e-45cd-ac7d-6e9e3625aaf6-webhook-cert\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.996313 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.996286 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a545c090-555e-45cd-ac7d-6e9e3625aaf6-tmp\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:17.997880 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:17.997776 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a545c090-555e-45cd-ac7d-6e9e3625aaf6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:18.001928 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:18.001903 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b69nr\" (UniqueName: \"kubernetes.io/projected/a545c090-555e-45cd-ac7d-6e9e3625aaf6-kube-api-access-b69nr\") pod \"opendatahub-operator-controller-manager-9fc4f56d7-lnd8l\" (UID: \"a545c090-555e-45cd-ac7d-6e9e3625aaf6\") " pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:18.120097 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:18.120062 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:18.221172 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:18.221109 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx"] Sep 05 07:53:18.223788 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:53:18.223758 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabdb768_8330_431b_ab10_a54f863c7500.slice/crio-adf87084f372bb4f39683c3862f35d72bb278a0863a528d606cf56b457625e02 WatchSource:0}: Error finding container adf87084f372bb4f39683c3862f35d72bb278a0863a528d606cf56b457625e02: Status 404 returned error can't find the container with id adf87084f372bb4f39683c3862f35d72bb278a0863a528d606cf56b457625e02 Sep 05 07:53:18.245030 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:18.244997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" event={"ID":"dabdb768-8330-431b-ab10-a54f863c7500","Type":"ContainerStarted","Data":"adf87084f372bb4f39683c3862f35d72bb278a0863a528d606cf56b457625e02"} Sep 05 07:53:18.409582 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:18.409544 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l"] Sep 05 07:53:18.411780 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:53:18.411758 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda545c090_555e_45cd_ac7d_6e9e3625aaf6.slice/crio-d46e93394c00300083905bd26a0c493c4f2668a486f96db83bf6c85ba64fe074 WatchSource:0}: Error finding container d46e93394c00300083905bd26a0c493c4f2668a486f96db83bf6c85ba64fe074: Status 404 returned error can't find the container with id d46e93394c00300083905bd26a0c493c4f2668a486f96db83bf6c85ba64fe074 Sep 05 07:53:19.253134 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:19.253091 2576 generic.go:356] "Generic (PLEG): container finished" podID="dabdb768-8330-431b-ab10-a54f863c7500" containerID="7131b07877a725dcbe81b6bf85c10855b26d63d47203f3cadf0d1bc4877f5b7c" exitCode=0 Sep 05 07:53:19.253635 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:19.253269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" event={"ID":"dabdb768-8330-431b-ab10-a54f863c7500","Type":"ContainerDied","Data":"7131b07877a725dcbe81b6bf85c10855b26d63d47203f3cadf0d1bc4877f5b7c"} Sep 05 07:53:19.254081 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:19.254002 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 07:53:19.255694 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:19.255660 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" event={"ID":"a545c090-555e-45cd-ac7d-6e9e3625aaf6","Type":"ContainerStarted","Data":"d46e93394c00300083905bd26a0c493c4f2668a486f96db83bf6c85ba64fe074"} Sep 05 07:53:22.272454 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:22.272415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" event={"ID":"a545c090-555e-45cd-ac7d-6e9e3625aaf6","Type":"ContainerStarted","Data":"d7179ee7c6c831b8ae39742a5d122e7705e439fc76c37343fc04c3b3fdd3bcd4"} Sep 05 07:53:22.274195 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:22.274169 2576 generic.go:356] "Generic (PLEG): container finished" podID="dabdb768-8330-431b-ab10-a54f863c7500" containerID="f9143a04c642e4854a0d49a09dab7d3d95329000e0bedb6939afde19656b0522" exitCode=0 Sep 05 07:53:22.274334 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:22.274257 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" event={"ID":"dabdb768-8330-431b-ab10-a54f863c7500","Type":"ContainerDied","Data":"f9143a04c642e4854a0d49a09dab7d3d95329000e0bedb6939afde19656b0522"} Sep 05 07:53:23.281314 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:23.281276 2576 generic.go:356] "Generic (PLEG): container finished" podID="dabdb768-8330-431b-ab10-a54f863c7500" containerID="ff181809ce088d618789af7905c64e2be8c44bb41fada2494465ae2328b2f945" exitCode=0 Sep 05 07:53:23.281727 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:23.281354 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" event={"ID":"dabdb768-8330-431b-ab10-a54f863c7500","Type":"ContainerDied","Data":"ff181809ce088d618789af7905c64e2be8c44bb41fada2494465ae2328b2f945"} Sep 05 07:53:23.282995 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:23.282969 2576 generic.go:356] "Generic (PLEG): container finished" podID="a545c090-555e-45cd-ac7d-6e9e3625aaf6" containerID="d7179ee7c6c831b8ae39742a5d122e7705e439fc76c37343fc04c3b3fdd3bcd4" exitCode=0 Sep 05 07:53:23.282995 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:23.283004 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" event={"ID":"a545c090-555e-45cd-ac7d-6e9e3625aaf6","Type":"ContainerDied","Data":"d7179ee7c6c831b8ae39742a5d122e7705e439fc76c37343fc04c3b3fdd3bcd4"} Sep 05 07:53:24.289295 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:24.289257 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" event={"ID":"a545c090-555e-45cd-ac7d-6e9e3625aaf6","Type":"ContainerStarted","Data":"f9c4720393d3a1d8c0d47a92cec8bd66e55df6c5120375070ef5601faeb46579"} Sep 05 07:53:24.309733 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:24.309686 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" podStartSLOduration=2.636810027 podStartE2EDuration="7.309667815s" totalImagesPullingTime="3.497818491s" totalInitContainerRuntime="1.175039297s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:53:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:53:24.308926552 +0000 UTC m=+306.917901676" watchObservedRunningTime="2026-09-05 07:53:24.309667815 +0000 UTC m=+306.918642938" Sep 05 07:53:24.417080 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:24.417056 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" Sep 05 07:53:24.444560 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:24.444525 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dabdb768-8330-431b-ab10-a54f863c7500-bundle\") pod \"dabdb768-8330-431b-ab10-a54f863c7500\" (UID: \"dabdb768-8330-431b-ab10-a54f863c7500\") " Sep 05 07:53:24.444682 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:24.444623 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dabdb768-8330-431b-ab10-a54f863c7500-util\") pod \"dabdb768-8330-431b-ab10-a54f863c7500\" (UID: \"dabdb768-8330-431b-ab10-a54f863c7500\") " Sep 05 07:53:24.444682 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:24.444662 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfxt6\" (UniqueName: \"kubernetes.io/projected/dabdb768-8330-431b-ab10-a54f863c7500-kube-api-access-xfxt6\") pod \"dabdb768-8330-431b-ab10-a54f863c7500\" (UID: \"dabdb768-8330-431b-ab10-a54f863c7500\") " Sep 05 07:53:24.445318 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:24.445290 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabdb768-8330-431b-ab10-a54f863c7500-bundle" (OuterVolumeSpecName: "bundle") pod "dabdb768-8330-431b-ab10-a54f863c7500" (UID: "dabdb768-8330-431b-ab10-a54f863c7500"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:53:24.446922 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:24.446898 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabdb768-8330-431b-ab10-a54f863c7500-kube-api-access-xfxt6" (OuterVolumeSpecName: "kube-api-access-xfxt6") pod "dabdb768-8330-431b-ab10-a54f863c7500" (UID: "dabdb768-8330-431b-ab10-a54f863c7500"). InnerVolumeSpecName "kube-api-access-xfxt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:53:24.450468 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:24.450423 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dabdb768-8330-431b-ab10-a54f863c7500-util" (OuterVolumeSpecName: "util") pod "dabdb768-8330-431b-ab10-a54f863c7500" (UID: "dabdb768-8330-431b-ab10-a54f863c7500"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:53:24.546112 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:24.546023 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dabdb768-8330-431b-ab10-a54f863c7500-bundle\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:53:24.546112 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:24.546052 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dabdb768-8330-431b-ab10-a54f863c7500-util\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:53:24.546112 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:24.546067 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfxt6\" (UniqueName: \"kubernetes.io/projected/dabdb768-8330-431b-ab10-a54f863c7500-kube-api-access-xfxt6\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:53:25.294967 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:25.294935 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" Sep 05 07:53:25.295404 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:25.294965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0rbzjx" event={"ID":"dabdb768-8330-431b-ab10-a54f863c7500","Type":"ContainerDied","Data":"adf87084f372bb4f39683c3862f35d72bb278a0863a528d606cf56b457625e02"} Sep 05 07:53:25.295404 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:25.295017 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adf87084f372bb4f39683c3862f35d72bb278a0863a528d606cf56b457625e02" Sep 05 07:53:25.295404 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:25.295395 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:36.302007 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:36.301972 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9fc4f56d7-lnd8l" Sep 05 07:53:38.448507 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.448471 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg"] Sep 05 07:53:38.448903 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.448732 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dabdb768-8330-431b-ab10-a54f863c7500" containerName="pull" Sep 05 07:53:38.448903 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.448742 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dabdb768-8330-431b-ab10-a54f863c7500" containerName="pull" Sep 05 07:53:38.448903 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.448751 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dabdb768-8330-431b-ab10-a54f863c7500" containerName="util" Sep 05 07:53:38.448903 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.448756 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dabdb768-8330-431b-ab10-a54f863c7500" containerName="util" Sep 05 07:53:38.448903 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.448775 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dabdb768-8330-431b-ab10-a54f863c7500" containerName="extract" Sep 05 07:53:38.448903 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.448780 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dabdb768-8330-431b-ab10-a54f863c7500" containerName="extract" Sep 05 07:53:38.448903 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.448823 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dabdb768-8330-431b-ab10-a54f863c7500" containerName="extract" Sep 05 07:53:38.451709 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.451688 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" Sep 05 07:53:38.454557 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.454535 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 05 07:53:38.454557 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.454546 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7jsrr\"" Sep 05 07:53:38.454749 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.454535 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 05 07:53:38.454749 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.454534 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 05 07:53:38.455183 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.455168 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 05 07:53:38.467823 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.467799 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg"] Sep 05 07:53:38.555094 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.555056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zch9n\" (UniqueName: \"kubernetes.io/projected/66cc75f7-944e-4f09-83ba-0e2c5f83221b-kube-api-access-zch9n\") pod \"kube-auth-proxy-74c76cbcf8-6mjdg\" (UID: \"66cc75f7-944e-4f09-83ba-0e2c5f83221b\") " pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" Sep 05 07:53:38.555094 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.555100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66cc75f7-944e-4f09-83ba-0e2c5f83221b-tmp\") pod \"kube-auth-proxy-74c76cbcf8-6mjdg\" (UID: \"66cc75f7-944e-4f09-83ba-0e2c5f83221b\") " pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" Sep 05 07:53:38.555337 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.555199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cc75f7-944e-4f09-83ba-0e2c5f83221b-tls-certs\") pod \"kube-auth-proxy-74c76cbcf8-6mjdg\" (UID: \"66cc75f7-944e-4f09-83ba-0e2c5f83221b\") " pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" Sep 05 07:53:38.656253 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.656199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cc75f7-944e-4f09-83ba-0e2c5f83221b-tls-certs\") pod \"kube-auth-proxy-74c76cbcf8-6mjdg\" (UID: \"66cc75f7-944e-4f09-83ba-0e2c5f83221b\") " pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" Sep 05 07:53:38.656421 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.656265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zch9n\" (UniqueName: \"kubernetes.io/projected/66cc75f7-944e-4f09-83ba-0e2c5f83221b-kube-api-access-zch9n\") pod \"kube-auth-proxy-74c76cbcf8-6mjdg\" (UID: \"66cc75f7-944e-4f09-83ba-0e2c5f83221b\") " pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" Sep 05 07:53:38.656421 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.656287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66cc75f7-944e-4f09-83ba-0e2c5f83221b-tmp\") pod \"kube-auth-proxy-74c76cbcf8-6mjdg\" (UID: \"66cc75f7-944e-4f09-83ba-0e2c5f83221b\") " pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" Sep 05 07:53:38.658699 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.658663 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66cc75f7-944e-4f09-83ba-0e2c5f83221b-tmp\") pod \"kube-auth-proxy-74c76cbcf8-6mjdg\" (UID: \"66cc75f7-944e-4f09-83ba-0e2c5f83221b\") " pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" Sep 05 07:53:38.658824 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.658775 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66cc75f7-944e-4f09-83ba-0e2c5f83221b-tls-certs\") pod \"kube-auth-proxy-74c76cbcf8-6mjdg\" (UID: \"66cc75f7-944e-4f09-83ba-0e2c5f83221b\") " pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" Sep 05 07:53:38.666194 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.666172 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zch9n\" (UniqueName: \"kubernetes.io/projected/66cc75f7-944e-4f09-83ba-0e2c5f83221b-kube-api-access-zch9n\") pod \"kube-auth-proxy-74c76cbcf8-6mjdg\" (UID: \"66cc75f7-944e-4f09-83ba-0e2c5f83221b\") " pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" Sep 05 07:53:38.761533 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:38.761433 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" Sep 05 07:53:39.035047 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.034971 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg"] Sep 05 07:53:39.037991 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:53:39.037960 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66cc75f7_944e_4f09_83ba_0e2c5f83221b.slice/crio-db987fcb48ea3627a59358831dc1508f7f3f7c3ffeec077fff023bd63378c230 WatchSource:0}: Error finding container db987fcb48ea3627a59358831dc1508f7f3f7c3ffeec077fff023bd63378c230: Status 404 returned error can't find the container with id db987fcb48ea3627a59358831dc1508f7f3f7c3ffeec077fff023bd63378c230 Sep 05 07:53:39.355924 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.355885 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" event={"ID":"66cc75f7-944e-4f09-83ba-0e2c5f83221b","Type":"ContainerStarted","Data":"db987fcb48ea3627a59358831dc1508f7f3f7c3ffeec077fff023bd63378c230"} Sep 05 07:53:39.843269 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.843238 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp"] Sep 05 07:53:39.846712 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.846689 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:39.849163 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.849136 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 05 07:53:39.849163 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.849156 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 05 07:53:39.849355 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.849179 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 05 07:53:39.849405 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.849379 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wr58z\"" Sep 05 07:53:39.849468 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.849452 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 05 07:53:39.860842 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.860818 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp"] Sep 05 07:53:39.967328 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.967282 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:39.967518 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.967350 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0bd75ae-9625-4342-bb5d-1cc804fda352-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:39.967518 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.967386 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:39.967518 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.967412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:39.967518 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.967433 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgt74\" (UniqueName: \"kubernetes.io/projected/d0bd75ae-9625-4342-bb5d-1cc804fda352-kube-api-access-cgt74\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:39.967733 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.967521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:39.967733 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:39.967602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.068799 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.068762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0bd75ae-9625-4342-bb5d-1cc804fda352-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.069004 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.068816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.069004 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.068850 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.069004 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.068881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgt74\" (UniqueName: \"kubernetes.io/projected/d0bd75ae-9625-4342-bb5d-1cc804fda352-kube-api-access-cgt74\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.069004 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.068922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.069004 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.068976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.069302 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.069018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.069614 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.069559 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.072164 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.072116 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.073309 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.073263 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.073924 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.073896 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0bd75ae-9625-4342-bb5d-1cc804fda352-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.074095 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.074067 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.078576 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.078550 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.078576 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.078566 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgt74\" (UniqueName: \"kubernetes.io/projected/d0bd75ae-9625-4342-bb5d-1cc804fda352-kube-api-access-cgt74\") pod \"istiod-openshift-gateway-7b7995d5cd-ktgjp\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.158774 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.158680 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:40.452396 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:40.452364 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp"] Sep 05 07:53:41.748068 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:53:41.748035 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bd75ae_9625_4342_bb5d_1cc804fda352.slice/crio-65a198d5b9ec01f5daf5d3b9dfbbbe14a1fc081f3ef8e11b2e26da05c42a3c35 WatchSource:0}: Error finding container 65a198d5b9ec01f5daf5d3b9dfbbbe14a1fc081f3ef8e11b2e26da05c42a3c35: Status 404 returned error can't find the container with id 65a198d5b9ec01f5daf5d3b9dfbbbe14a1fc081f3ef8e11b2e26da05c42a3c35 Sep 05 07:53:42.371626 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:42.371577 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" event={"ID":"d0bd75ae-9625-4342-bb5d-1cc804fda352","Type":"ContainerStarted","Data":"65a198d5b9ec01f5daf5d3b9dfbbbe14a1fc081f3ef8e11b2e26da05c42a3c35"} Sep 05 07:53:42.373082 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:42.373053 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" event={"ID":"66cc75f7-944e-4f09-83ba-0e2c5f83221b","Type":"ContainerStarted","Data":"2600e4e52ced57e04f68115a25fceca86be1744dbf006eff1ac10e4f660e979f"} Sep 05 07:53:42.389796 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:42.389748 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-74c76cbcf8-6mjdg" podStartSLOduration=1.6225835709999998 podStartE2EDuration="4.389729349s" totalImagesPullingTime="2.767145778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:53:42.388694772 +0000 UTC m=+324.997669896" watchObservedRunningTime="2026-09-05 07:53:42.389729349 +0000 UTC m=+324.998704471" Sep 05 07:53:44.305430 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:44.305380 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 05 07:53:44.305728 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:44.305455 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 05 07:53:44.385912 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:44.385809 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" event={"ID":"d0bd75ae-9625-4342-bb5d-1cc804fda352","Type":"ContainerStarted","Data":"2f199407337a43176d77ae74c5e2e26b9c52fbe08b923d18ddecd05c7f78563f"} Sep 05 07:53:44.385912 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:44.385882 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:44.406982 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:44.406930 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" podStartSLOduration=2.8518743779999998 podStartE2EDuration="5.406913198s" totalImagesPullingTime="2.55503882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:53:44.406185905 +0000 UTC m=+327.015161028" watchObservedRunningTime="2026-09-05 07:53:44.406913198 +0000 UTC m=+327.015888321" Sep 05 07:53:45.392635 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:45.392601 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-ktgjp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 05 07:53:45.393036 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:45.392670 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" podUID="d0bd75ae-9625-4342-bb5d-1cc804fda352" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 05 07:53:48.391976 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:48.391944 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:56.182021 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.181942 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4"] Sep 05 07:53:56.189547 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.189520 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.197432 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.197408 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4"] Sep 05 07:53:56.308797 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.308752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t2br\" (UniqueName: \"kubernetes.io/projected/607f4444-997d-48a8-81d7-015547d579f6-kube-api-access-7t2br\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.308797 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.308808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/607f4444-997d-48a8-81d7-015547d579f6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.309015 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.308828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/607f4444-997d-48a8-81d7-015547d579f6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.309015 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.308853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/607f4444-997d-48a8-81d7-015547d579f6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.309015 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.308879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/607f4444-997d-48a8-81d7-015547d579f6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.309015 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.308923 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/607f4444-997d-48a8-81d7-015547d579f6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.309015 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.308937 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/607f4444-997d-48a8-81d7-015547d579f6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.410257 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.410194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/607f4444-997d-48a8-81d7-015547d579f6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.410454 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.410278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/607f4444-997d-48a8-81d7-015547d579f6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.410454 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.410330 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/607f4444-997d-48a8-81d7-015547d579f6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.410454 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.410381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/607f4444-997d-48a8-81d7-015547d579f6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.410454 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.410432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/607f4444-997d-48a8-81d7-015547d579f6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.410866 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.410839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/607f4444-997d-48a8-81d7-015547d579f6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.411011 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.410923 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t2br\" (UniqueName: \"kubernetes.io/projected/607f4444-997d-48a8-81d7-015547d579f6-kube-api-access-7t2br\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.411767 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.411743 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/607f4444-997d-48a8-81d7-015547d579f6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.413364 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.413334 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/607f4444-997d-48a8-81d7-015547d579f6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.413483 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.413365 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/607f4444-997d-48a8-81d7-015547d579f6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.413483 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.413436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/607f4444-997d-48a8-81d7-015547d579f6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.414395 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.414366 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/607f4444-997d-48a8-81d7-015547d579f6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.428971 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.424845 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/607f4444-997d-48a8-81d7-015547d579f6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.428971 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.426169 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t2br\" (UniqueName: \"kubernetes.io/projected/607f4444-997d-48a8-81d7-015547d579f6-kube-api-access-7t2br\") pod \"istiod-openshift-gateway-5677ccb49b-v8mm4\" (UID: \"607f4444-997d-48a8-81d7-015547d579f6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.499201 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.499104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:56.775967 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.775886 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4"] Sep 05 07:53:56.779525 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:53:56.779500 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod607f4444_997d_48a8_81d7_015547d579f6.slice/crio-64b336648bc16401eb50158fe29e8a89744e3f6bc294a74bddb12161f2caf323 WatchSource:0}: Error finding container 64b336648bc16401eb50158fe29e8a89744e3f6bc294a74bddb12161f2caf323: Status 404 returned error can't find the container with id 64b336648bc16401eb50158fe29e8a89744e3f6bc294a74bddb12161f2caf323 Sep 05 07:53:56.781560 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.781529 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 05 07:53:56.781643 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:56.781600 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 05 07:53:57.452685 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:57.452644 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" event={"ID":"607f4444-997d-48a8-81d7-015547d579f6","Type":"ContainerStarted","Data":"98908b40ea882763523379bcf44d6021d7b360aea13cfb5573943b5d33339e5f"} Sep 05 07:53:57.453190 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:57.452735 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" event={"ID":"607f4444-997d-48a8-81d7-015547d579f6","Type":"ContainerStarted","Data":"64b336648bc16401eb50158fe29e8a89744e3f6bc294a74bddb12161f2caf323"} Sep 05 07:53:57.453190 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:57.452762 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:57.454649 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:57.454620 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-v8mm4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 05 07:53:57.454753 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:57.454670 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" podUID="607f4444-997d-48a8-81d7-015547d579f6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 05 07:53:57.481667 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:57.481610 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" podStartSLOduration=1.481591186 podStartE2EDuration="1.481591186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:53:57.48041567 +0000 UTC m=+340.089390794" watchObservedRunningTime="2026-09-05 07:53:57.481591186 +0000 UTC m=+340.090566308" Sep 05 07:53:58.457356 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:58.457325 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-v8mm4" Sep 05 07:53:58.595403 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:58.595369 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp"] Sep 05 07:53:58.595607 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:58.595584 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" podUID="d0bd75ae-9625-4342-bb5d-1cc804fda352" containerName="discovery" containerID="cri-o://2f199407337a43176d77ae74c5e2e26b9c52fbe08b923d18ddecd05c7f78563f" gracePeriod=30 Sep 05 07:53:58.857204 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:58.857175 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:59.032199 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.032112 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgt74\" (UniqueName: \"kubernetes.io/projected/d0bd75ae-9625-4342-bb5d-1cc804fda352-kube-api-access-cgt74\") pod \"d0bd75ae-9625-4342-bb5d-1cc804fda352\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " Sep 05 07:53:59.032199 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.032185 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-token\") pod \"d0bd75ae-9625-4342-bb5d-1cc804fda352\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " Sep 05 07:53:59.032453 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.032241 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0bd75ae-9625-4342-bb5d-1cc804fda352-local-certs\") pod \"d0bd75ae-9625-4342-bb5d-1cc804fda352\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " Sep 05 07:53:59.032453 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.032263 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-csr-ca-configmap\") pod \"d0bd75ae-9625-4342-bb5d-1cc804fda352\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " Sep 05 07:53:59.032453 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.032291 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-kubeconfig\") pod \"d0bd75ae-9625-4342-bb5d-1cc804fda352\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " Sep 05 07:53:59.032901 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.032868 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d0bd75ae-9625-4342-bb5d-1cc804fda352" (UID: "d0bd75ae-9625-4342-bb5d-1cc804fda352"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 07:53:59.033085 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.032937 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-csr-dns-cert\") pod \"d0bd75ae-9625-4342-bb5d-1cc804fda352\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " Sep 05 07:53:59.033085 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.032975 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-cacerts\") pod \"d0bd75ae-9625-4342-bb5d-1cc804fda352\" (UID: \"d0bd75ae-9625-4342-bb5d-1cc804fda352\") " Sep 05 07:53:59.033085 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.033247 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-csr-ca-configmap\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:53:59.035362 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.035335 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0bd75ae-9625-4342-bb5d-1cc804fda352-local-certs" (OuterVolumeSpecName: "local-certs") pod "d0bd75ae-9625-4342-bb5d-1cc804fda352" (UID: "d0bd75ae-9625-4342-bb5d-1cc804fda352"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:53:59.035631 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.035599 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bd75ae-9625-4342-bb5d-1cc804fda352-kube-api-access-cgt74" (OuterVolumeSpecName: "kube-api-access-cgt74") pod "d0bd75ae-9625-4342-bb5d-1cc804fda352" (UID: "d0bd75ae-9625-4342-bb5d-1cc804fda352"). InnerVolumeSpecName "kube-api-access-cgt74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:53:59.035764 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.035708 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d0bd75ae-9625-4342-bb5d-1cc804fda352" (UID: "d0bd75ae-9625-4342-bb5d-1cc804fda352"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 07:53:59.035999 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.035963 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-token" (OuterVolumeSpecName: "istio-token") pod "d0bd75ae-9625-4342-bb5d-1cc804fda352" (UID: "d0bd75ae-9625-4342-bb5d-1cc804fda352"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:53:59.036327 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.036306 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-cacerts" (OuterVolumeSpecName: "cacerts") pod "d0bd75ae-9625-4342-bb5d-1cc804fda352" (UID: "d0bd75ae-9625-4342-bb5d-1cc804fda352"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 07:53:59.036529 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.036505 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d0bd75ae-9625-4342-bb5d-1cc804fda352" (UID: "d0bd75ae-9625-4342-bb5d-1cc804fda352"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 07:53:59.134198 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.134144 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgt74\" (UniqueName: \"kubernetes.io/projected/d0bd75ae-9625-4342-bb5d-1cc804fda352-kube-api-access-cgt74\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:53:59.134198 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.134191 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-token\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:53:59.134198 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.134201 2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0bd75ae-9625-4342-bb5d-1cc804fda352-local-certs\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:53:59.134198 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.134237 2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-kubeconfig\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:53:59.134512 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.134246 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-istio-csr-dns-cert\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:53:59.134512 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.134255 2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d0bd75ae-9625-4342-bb5d-1cc804fda352-cacerts\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:53:59.462679 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.462641 2576 generic.go:356] "Generic (PLEG): container finished" podID="d0bd75ae-9625-4342-bb5d-1cc804fda352" containerID="2f199407337a43176d77ae74c5e2e26b9c52fbe08b923d18ddecd05c7f78563f" exitCode=0 Sep 05 07:53:59.463119 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.462686 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" Sep 05 07:53:59.463119 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.462725 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" event={"ID":"d0bd75ae-9625-4342-bb5d-1cc804fda352","Type":"ContainerDied","Data":"2f199407337a43176d77ae74c5e2e26b9c52fbe08b923d18ddecd05c7f78563f"} Sep 05 07:53:59.463119 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.462777 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp" event={"ID":"d0bd75ae-9625-4342-bb5d-1cc804fda352","Type":"ContainerDied","Data":"65a198d5b9ec01f5daf5d3b9dfbbbe14a1fc081f3ef8e11b2e26da05c42a3c35"} Sep 05 07:53:59.463119 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.462798 2576 scope.go:117] "RemoveContainer" containerID="2f199407337a43176d77ae74c5e2e26b9c52fbe08b923d18ddecd05c7f78563f" Sep 05 07:53:59.471786 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.471764 2576 scope.go:117] "RemoveContainer" containerID="2f199407337a43176d77ae74c5e2e26b9c52fbe08b923d18ddecd05c7f78563f" Sep 05 07:53:59.472111 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:53:59.472093 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f199407337a43176d77ae74c5e2e26b9c52fbe08b923d18ddecd05c7f78563f\": container with ID starting with 2f199407337a43176d77ae74c5e2e26b9c52fbe08b923d18ddecd05c7f78563f not found: ID does not exist" containerID="2f199407337a43176d77ae74c5e2e26b9c52fbe08b923d18ddecd05c7f78563f" Sep 05 07:53:59.472160 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.472121 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f199407337a43176d77ae74c5e2e26b9c52fbe08b923d18ddecd05c7f78563f"} err="failed to get container status \"2f199407337a43176d77ae74c5e2e26b9c52fbe08b923d18ddecd05c7f78563f\": rpc error: code = NotFound desc = could not find container \"2f199407337a43176d77ae74c5e2e26b9c52fbe08b923d18ddecd05c7f78563f\": container with ID starting with 2f199407337a43176d77ae74c5e2e26b9c52fbe08b923d18ddecd05c7f78563f not found: ID does not exist" Sep 05 07:53:59.490107 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.490081 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp"] Sep 05 07:53:59.493563 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.493540 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-ktgjp"] Sep 05 07:53:59.885927 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:53:59.885896 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0bd75ae-9625-4342-bb5d-1cc804fda352" path="/var/lib/kubelet/pods/d0bd75ae-9625-4342-bb5d-1cc804fda352/volumes" Sep 05 07:54:53.758738 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.758703 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7"] Sep 05 07:54:53.759253 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.758986 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0bd75ae-9625-4342-bb5d-1cc804fda352" containerName="discovery" Sep 05 07:54:53.759253 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.758997 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0bd75ae-9625-4342-bb5d-1cc804fda352" containerName="discovery" Sep 05 07:54:53.759253 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.759053 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0bd75ae-9625-4342-bb5d-1cc804fda352" containerName="discovery" Sep 05 07:54:53.762002 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.761988 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" Sep 05 07:54:53.764009 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.763991 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-cns7m\"" Sep 05 07:54:53.764009 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.764002 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 05 07:54:53.764501 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.764487 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 05 07:54:53.770148 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.770127 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7"] Sep 05 07:54:53.846503 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.846474 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xxzc\" (UniqueName: \"kubernetes.io/projected/9be19cc8-3304-4522-a606-7652d9014932-kube-api-access-9xxzc\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7\" (UID: \"9be19cc8-3304-4522-a606-7652d9014932\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" Sep 05 07:54:53.846658 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.846522 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9be19cc8-3304-4522-a606-7652d9014932-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7\" (UID: \"9be19cc8-3304-4522-a606-7652d9014932\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" Sep 05 07:54:53.846658 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.846597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9be19cc8-3304-4522-a606-7652d9014932-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7\" (UID: \"9be19cc8-3304-4522-a606-7652d9014932\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" Sep 05 07:54:53.853301 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.853277 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f"] Sep 05 07:54:53.856390 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.856376 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" Sep 05 07:54:53.864746 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.864726 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f"] Sep 05 07:54:53.947391 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.947353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9be19cc8-3304-4522-a606-7652d9014932-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7\" (UID: \"9be19cc8-3304-4522-a606-7652d9014932\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" Sep 05 07:54:53.947560 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.947398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef28f64b-9d52-441c-89b3-1a81294a172b-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f\" (UID: \"ef28f64b-9d52-441c-89b3-1a81294a172b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" Sep 05 07:54:53.947560 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.947452 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef28f64b-9d52-441c-89b3-1a81294a172b-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f\" (UID: \"ef28f64b-9d52-441c-89b3-1a81294a172b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" Sep 05 07:54:53.947560 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.947506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9be19cc8-3304-4522-a606-7652d9014932-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7\" (UID: \"9be19cc8-3304-4522-a606-7652d9014932\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" Sep 05 07:54:53.947698 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.947561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8xm5\" (UniqueName: \"kubernetes.io/projected/ef28f64b-9d52-441c-89b3-1a81294a172b-kube-api-access-d8xm5\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f\" (UID: \"ef28f64b-9d52-441c-89b3-1a81294a172b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" Sep 05 07:54:53.947698 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.947585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xxzc\" (UniqueName: \"kubernetes.io/projected/9be19cc8-3304-4522-a606-7652d9014932-kube-api-access-9xxzc\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7\" (UID: \"9be19cc8-3304-4522-a606-7652d9014932\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" Sep 05 07:54:53.947815 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.947796 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9be19cc8-3304-4522-a606-7652d9014932-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7\" (UID: \"9be19cc8-3304-4522-a606-7652d9014932\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" Sep 05 07:54:53.947878 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.947837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9be19cc8-3304-4522-a606-7652d9014932-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7\" (UID: \"9be19cc8-3304-4522-a606-7652d9014932\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" Sep 05 07:54:53.958500 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.958474 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn"] Sep 05 07:54:53.960565 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.960544 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xxzc\" (UniqueName: \"kubernetes.io/projected/9be19cc8-3304-4522-a606-7652d9014932-kube-api-access-9xxzc\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7\" (UID: \"9be19cc8-3304-4522-a606-7652d9014932\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" Sep 05 07:54:53.967071 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.967051 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" Sep 05 07:54:53.969938 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:53.969904 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn"] Sep 05 07:54:54.048064 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.047992 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e054e24-b4cf-43ec-ba86-01cd905083b8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn\" (UID: \"6e054e24-b4cf-43ec-ba86-01cd905083b8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" Sep 05 07:54:54.048064 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.048020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e054e24-b4cf-43ec-ba86-01cd905083b8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn\" (UID: \"6e054e24-b4cf-43ec-ba86-01cd905083b8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" Sep 05 07:54:54.048064 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.048051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef28f64b-9d52-441c-89b3-1a81294a172b-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f\" (UID: \"ef28f64b-9d52-441c-89b3-1a81294a172b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" Sep 05 07:54:54.048305 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.048197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef28f64b-9d52-441c-89b3-1a81294a172b-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f\" (UID: \"ef28f64b-9d52-441c-89b3-1a81294a172b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" Sep 05 07:54:54.048305 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.048245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8xm5\" (UniqueName: \"kubernetes.io/projected/ef28f64b-9d52-441c-89b3-1a81294a172b-kube-api-access-d8xm5\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f\" (UID: \"ef28f64b-9d52-441c-89b3-1a81294a172b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" Sep 05 07:54:54.048305 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.048266 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2t9q\" (UniqueName: \"kubernetes.io/projected/6e054e24-b4cf-43ec-ba86-01cd905083b8-kube-api-access-n2t9q\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn\" (UID: \"6e054e24-b4cf-43ec-ba86-01cd905083b8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" Sep 05 07:54:54.048420 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.048361 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef28f64b-9d52-441c-89b3-1a81294a172b-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f\" (UID: \"ef28f64b-9d52-441c-89b3-1a81294a172b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" Sep 05 07:54:54.048524 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.048507 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef28f64b-9d52-441c-89b3-1a81294a172b-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f\" (UID: \"ef28f64b-9d52-441c-89b3-1a81294a172b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" Sep 05 07:54:54.057289 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.057270 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8xm5\" (UniqueName: \"kubernetes.io/projected/ef28f64b-9d52-441c-89b3-1a81294a172b-kube-api-access-d8xm5\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f\" (UID: \"ef28f64b-9d52-441c-89b3-1a81294a172b\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" Sep 05 07:54:54.059571 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.059551 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l"] Sep 05 07:54:54.062855 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.062834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" Sep 05 07:54:54.070687 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.070666 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l"] Sep 05 07:54:54.070784 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.070775 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" Sep 05 07:54:54.148859 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.148831 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad6ae678-c862-41c4-b393-404f53fd97cd-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l\" (UID: \"ad6ae678-c862-41c4-b393-404f53fd97cd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" Sep 05 07:54:54.148995 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.148867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24pd4\" (UniqueName: \"kubernetes.io/projected/ad6ae678-c862-41c4-b393-404f53fd97cd-kube-api-access-24pd4\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l\" (UID: \"ad6ae678-c862-41c4-b393-404f53fd97cd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" Sep 05 07:54:54.148995 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.148917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad6ae678-c862-41c4-b393-404f53fd97cd-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l\" (UID: \"ad6ae678-c862-41c4-b393-404f53fd97cd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" Sep 05 07:54:54.148995 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.148957 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2t9q\" (UniqueName: \"kubernetes.io/projected/6e054e24-b4cf-43ec-ba86-01cd905083b8-kube-api-access-n2t9q\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn\" (UID: \"6e054e24-b4cf-43ec-ba86-01cd905083b8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" Sep 05 07:54:54.149106 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.149014 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e054e24-b4cf-43ec-ba86-01cd905083b8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn\" (UID: \"6e054e24-b4cf-43ec-ba86-01cd905083b8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" Sep 05 07:54:54.149106 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.149039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e054e24-b4cf-43ec-ba86-01cd905083b8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn\" (UID: \"6e054e24-b4cf-43ec-ba86-01cd905083b8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" Sep 05 07:54:54.149433 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.149417 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e054e24-b4cf-43ec-ba86-01cd905083b8-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn\" (UID: \"6e054e24-b4cf-43ec-ba86-01cd905083b8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" Sep 05 07:54:54.149493 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.149475 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e054e24-b4cf-43ec-ba86-01cd905083b8-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn\" (UID: \"6e054e24-b4cf-43ec-ba86-01cd905083b8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" Sep 05 07:54:54.157437 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.157414 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2t9q\" (UniqueName: \"kubernetes.io/projected/6e054e24-b4cf-43ec-ba86-01cd905083b8-kube-api-access-n2t9q\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn\" (UID: \"6e054e24-b4cf-43ec-ba86-01cd905083b8\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" Sep 05 07:54:54.166243 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.166206 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" Sep 05 07:54:54.249898 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.249867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad6ae678-c862-41c4-b393-404f53fd97cd-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l\" (UID: \"ad6ae678-c862-41c4-b393-404f53fd97cd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" Sep 05 07:54:54.250046 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.249940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad6ae678-c862-41c4-b393-404f53fd97cd-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l\" (UID: \"ad6ae678-c862-41c4-b393-404f53fd97cd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" Sep 05 07:54:54.250046 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.249961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24pd4\" (UniqueName: \"kubernetes.io/projected/ad6ae678-c862-41c4-b393-404f53fd97cd-kube-api-access-24pd4\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l\" (UID: \"ad6ae678-c862-41c4-b393-404f53fd97cd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" Sep 05 07:54:54.250272 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.250250 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad6ae678-c862-41c4-b393-404f53fd97cd-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l\" (UID: \"ad6ae678-c862-41c4-b393-404f53fd97cd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" Sep 05 07:54:54.250350 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.250304 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad6ae678-c862-41c4-b393-404f53fd97cd-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l\" (UID: \"ad6ae678-c862-41c4-b393-404f53fd97cd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" Sep 05 07:54:54.258011 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.257989 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24pd4\" (UniqueName: \"kubernetes.io/projected/ad6ae678-c862-41c4-b393-404f53fd97cd-kube-api-access-24pd4\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l\" (UID: \"ad6ae678-c862-41c4-b393-404f53fd97cd\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" Sep 05 07:54:54.276499 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.276478 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" Sep 05 07:54:54.346306 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.346271 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7"] Sep 05 07:54:54.350409 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:54:54.350378 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be19cc8_3304_4522_a606_7652d9014932.slice/crio-29266f5a1d7ef1beada0046d375fced2a90d8273f5580c44fb09248bd6c07317 WatchSource:0}: Error finding container 29266f5a1d7ef1beada0046d375fced2a90d8273f5580c44fb09248bd6c07317: Status 404 returned error can't find the container with id 29266f5a1d7ef1beada0046d375fced2a90d8273f5580c44fb09248bd6c07317 Sep 05 07:54:54.372428 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.372408 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" Sep 05 07:54:54.438305 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.438263 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f"] Sep 05 07:54:54.440654 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:54:54.440627 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef28f64b_9d52_441c_89b3_1a81294a172b.slice/crio-e6633cd062b53c53b077e00cc4a26fbf677ff36e8883bcfac40c7a1ad83f0d1d WatchSource:0}: Error finding container e6633cd062b53c53b077e00cc4a26fbf677ff36e8883bcfac40c7a1ad83f0d1d: Status 404 returned error can't find the container with id e6633cd062b53c53b077e00cc4a26fbf677ff36e8883bcfac40c7a1ad83f0d1d Sep 05 07:54:54.550078 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.550053 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn"] Sep 05 07:54:54.552528 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:54:54.552505 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e054e24_b4cf_43ec_ba86_01cd905083b8.slice/crio-b72c5b864618c10cd267a059e4b5c933fa51d0adfd0f976358ab1eedcc9b7206 WatchSource:0}: Error finding container b72c5b864618c10cd267a059e4b5c933fa51d0adfd0f976358ab1eedcc9b7206: Status 404 returned error can't find the container with id b72c5b864618c10cd267a059e4b5c933fa51d0adfd0f976358ab1eedcc9b7206 Sep 05 07:54:54.645975 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.645947 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l"] Sep 05 07:54:54.705047 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:54:54.705022 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad6ae678_c862_41c4_b393_404f53fd97cd.slice/crio-83eaa56e87776180edb2b7390a2946ce322a0c962b7604c0e2761127fc066ceb WatchSource:0}: Error finding container 83eaa56e87776180edb2b7390a2946ce322a0c962b7604c0e2761127fc066ceb: Status 404 returned error can't find the container with id 83eaa56e87776180edb2b7390a2946ce322a0c962b7604c0e2761127fc066ceb Sep 05 07:54:54.714162 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.714133 2576 generic.go:356] "Generic (PLEG): container finished" podID="ef28f64b-9d52-441c-89b3-1a81294a172b" containerID="d8e663e3e4889cb09d30c0708b0cd7364f5f0da36dad9a8061bf960dbc59d15a" exitCode=0 Sep 05 07:54:54.714271 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.714191 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" event={"ID":"ef28f64b-9d52-441c-89b3-1a81294a172b","Type":"ContainerDied","Data":"d8e663e3e4889cb09d30c0708b0cd7364f5f0da36dad9a8061bf960dbc59d15a"} Sep 05 07:54:54.714333 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.714270 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" event={"ID":"ef28f64b-9d52-441c-89b3-1a81294a172b","Type":"ContainerStarted","Data":"e6633cd062b53c53b077e00cc4a26fbf677ff36e8883bcfac40c7a1ad83f0d1d"} Sep 05 07:54:54.715794 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.715776 2576 generic.go:356] "Generic (PLEG): container finished" podID="9be19cc8-3304-4522-a606-7652d9014932" containerID="fcd287cfa90c189114a057a886294c59e1687baa6f7b7ae0799a8eaa6107753b" exitCode=0 Sep 05 07:54:54.715903 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.715861 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" event={"ID":"9be19cc8-3304-4522-a606-7652d9014932","Type":"ContainerDied","Data":"fcd287cfa90c189114a057a886294c59e1687baa6f7b7ae0799a8eaa6107753b"} Sep 05 07:54:54.715996 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.715909 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" event={"ID":"9be19cc8-3304-4522-a606-7652d9014932","Type":"ContainerStarted","Data":"29266f5a1d7ef1beada0046d375fced2a90d8273f5580c44fb09248bd6c07317"} Sep 05 07:54:54.717650 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.717630 2576 generic.go:356] "Generic (PLEG): container finished" podID="6e054e24-b4cf-43ec-ba86-01cd905083b8" containerID="14493db0569a0a8a84ad315fe37e4893b76b068428c9a710346212724d807259" exitCode=0 Sep 05 07:54:54.717744 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.717718 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" event={"ID":"6e054e24-b4cf-43ec-ba86-01cd905083b8","Type":"ContainerDied","Data":"14493db0569a0a8a84ad315fe37e4893b76b068428c9a710346212724d807259"} Sep 05 07:54:54.717806 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.717760 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" event={"ID":"6e054e24-b4cf-43ec-ba86-01cd905083b8","Type":"ContainerStarted","Data":"b72c5b864618c10cd267a059e4b5c933fa51d0adfd0f976358ab1eedcc9b7206"} Sep 05 07:54:54.719144 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:54.719128 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" event={"ID":"ad6ae678-c862-41c4-b393-404f53fd97cd","Type":"ContainerStarted","Data":"83eaa56e87776180edb2b7390a2946ce322a0c962b7604c0e2761127fc066ceb"} Sep 05 07:54:55.725351 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:55.725281 2576 generic.go:356] "Generic (PLEG): container finished" podID="ef28f64b-9d52-441c-89b3-1a81294a172b" containerID="d11c0f3f62c8e01f1b33bee82145a3468757fe4036484df3871cbbf75e9b1b98" exitCode=0 Sep 05 07:54:55.725682 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:55.725363 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" event={"ID":"ef28f64b-9d52-441c-89b3-1a81294a172b","Type":"ContainerDied","Data":"d11c0f3f62c8e01f1b33bee82145a3468757fe4036484df3871cbbf75e9b1b98"} Sep 05 07:54:55.727330 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:55.727311 2576 generic.go:356] "Generic (PLEG): container finished" podID="9be19cc8-3304-4522-a606-7652d9014932" containerID="72349c12122489e4a1b822c054d5ab7c557913adcf1709c7b01676ddabae320a" exitCode=0 Sep 05 07:54:55.727396 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:55.727379 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" event={"ID":"9be19cc8-3304-4522-a606-7652d9014932","Type":"ContainerDied","Data":"72349c12122489e4a1b822c054d5ab7c557913adcf1709c7b01676ddabae320a"} Sep 05 07:54:55.729563 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:55.729544 2576 generic.go:356] "Generic (PLEG): container finished" podID="6e054e24-b4cf-43ec-ba86-01cd905083b8" containerID="756331ef830911cf50dd0bc3015f579f8d49d7db0706a4060b07e14fbbea56eb" exitCode=0 Sep 05 07:54:55.729659 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:55.729629 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" event={"ID":"6e054e24-b4cf-43ec-ba86-01cd905083b8","Type":"ContainerDied","Data":"756331ef830911cf50dd0bc3015f579f8d49d7db0706a4060b07e14fbbea56eb"} Sep 05 07:54:55.731372 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:55.731353 2576 generic.go:356] "Generic (PLEG): container finished" podID="ad6ae678-c862-41c4-b393-404f53fd97cd" containerID="2809bee691f736ba61b8d90e632b9b00871df1b5db8bfa0fbb44caf22a1f94b6" exitCode=0 Sep 05 07:54:55.731439 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:55.731418 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" event={"ID":"ad6ae678-c862-41c4-b393-404f53fd97cd","Type":"ContainerDied","Data":"2809bee691f736ba61b8d90e632b9b00871df1b5db8bfa0fbb44caf22a1f94b6"} Sep 05 07:54:56.738358 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:56.738325 2576 generic.go:356] "Generic (PLEG): container finished" podID="ef28f64b-9d52-441c-89b3-1a81294a172b" containerID="cd2e972893f5bd4c71d80394103ed141f64ca72aafdd903d29fc94b8c097d6ab" exitCode=0 Sep 05 07:54:56.738751 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:56.738416 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" event={"ID":"ef28f64b-9d52-441c-89b3-1a81294a172b","Type":"ContainerDied","Data":"cd2e972893f5bd4c71d80394103ed141f64ca72aafdd903d29fc94b8c097d6ab"} Sep 05 07:54:56.740414 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:56.740390 2576 generic.go:356] "Generic (PLEG): container finished" podID="9be19cc8-3304-4522-a606-7652d9014932" containerID="95e4192c886ca6d4e920aa990d2b5cc6e06729fb9220ff743999ee2850033675" exitCode=0 Sep 05 07:54:56.740514 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:56.740483 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" event={"ID":"9be19cc8-3304-4522-a606-7652d9014932","Type":"ContainerDied","Data":"95e4192c886ca6d4e920aa990d2b5cc6e06729fb9220ff743999ee2850033675"} Sep 05 07:54:56.742404 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:56.742386 2576 generic.go:356] "Generic (PLEG): container finished" podID="6e054e24-b4cf-43ec-ba86-01cd905083b8" containerID="1dea810b37bf0b4322cfb279dc63fbf89fcbc7b15a4fe98d25bb3f1618bf273e" exitCode=0 Sep 05 07:54:56.742505 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:56.742445 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" event={"ID":"6e054e24-b4cf-43ec-ba86-01cd905083b8","Type":"ContainerDied","Data":"1dea810b37bf0b4322cfb279dc63fbf89fcbc7b15a4fe98d25bb3f1618bf273e"} Sep 05 07:54:57.903559 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.903476 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" Sep 05 07:54:57.906827 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.906806 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" Sep 05 07:54:57.952877 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.952850 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" Sep 05 07:54:57.980632 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.980609 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xxzc\" (UniqueName: \"kubernetes.io/projected/9be19cc8-3304-4522-a606-7652d9014932-kube-api-access-9xxzc\") pod \"9be19cc8-3304-4522-a606-7652d9014932\" (UID: \"9be19cc8-3304-4522-a606-7652d9014932\") " Sep 05 07:54:57.980758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.980690 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e054e24-b4cf-43ec-ba86-01cd905083b8-bundle\") pod \"6e054e24-b4cf-43ec-ba86-01cd905083b8\" (UID: \"6e054e24-b4cf-43ec-ba86-01cd905083b8\") " Sep 05 07:54:57.980758 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.980718 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9be19cc8-3304-4522-a606-7652d9014932-bundle\") pod \"9be19cc8-3304-4522-a606-7652d9014932\" (UID: \"9be19cc8-3304-4522-a606-7652d9014932\") " Sep 05 07:54:57.980864 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.980772 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2t9q\" (UniqueName: \"kubernetes.io/projected/6e054e24-b4cf-43ec-ba86-01cd905083b8-kube-api-access-n2t9q\") pod \"6e054e24-b4cf-43ec-ba86-01cd905083b8\" (UID: \"6e054e24-b4cf-43ec-ba86-01cd905083b8\") " Sep 05 07:54:57.980864 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.980800 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e054e24-b4cf-43ec-ba86-01cd905083b8-util\") pod \"6e054e24-b4cf-43ec-ba86-01cd905083b8\" (UID: \"6e054e24-b4cf-43ec-ba86-01cd905083b8\") " Sep 05 07:54:57.980864 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.980826 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9be19cc8-3304-4522-a606-7652d9014932-util\") pod \"9be19cc8-3304-4522-a606-7652d9014932\" (UID: \"9be19cc8-3304-4522-a606-7652d9014932\") " Sep 05 07:54:57.981775 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.981713 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be19cc8-3304-4522-a606-7652d9014932-bundle" (OuterVolumeSpecName: "bundle") pod "9be19cc8-3304-4522-a606-7652d9014932" (UID: "9be19cc8-3304-4522-a606-7652d9014932"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:54:57.982190 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.982164 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e054e24-b4cf-43ec-ba86-01cd905083b8-bundle" (OuterVolumeSpecName: "bundle") pod "6e054e24-b4cf-43ec-ba86-01cd905083b8" (UID: "6e054e24-b4cf-43ec-ba86-01cd905083b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:54:57.983340 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.983319 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be19cc8-3304-4522-a606-7652d9014932-kube-api-access-9xxzc" (OuterVolumeSpecName: "kube-api-access-9xxzc") pod "9be19cc8-3304-4522-a606-7652d9014932" (UID: "9be19cc8-3304-4522-a606-7652d9014932"). InnerVolumeSpecName "kube-api-access-9xxzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:54:57.984274 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.984252 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e054e24-b4cf-43ec-ba86-01cd905083b8-kube-api-access-n2t9q" (OuterVolumeSpecName: "kube-api-access-n2t9q") pod "6e054e24-b4cf-43ec-ba86-01cd905083b8" (UID: "6e054e24-b4cf-43ec-ba86-01cd905083b8"). InnerVolumeSpecName "kube-api-access-n2t9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:54:57.987166 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.987144 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be19cc8-3304-4522-a606-7652d9014932-util" (OuterVolumeSpecName: "util") pod "9be19cc8-3304-4522-a606-7652d9014932" (UID: "9be19cc8-3304-4522-a606-7652d9014932"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:54:57.990155 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:57.990119 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e054e24-b4cf-43ec-ba86-01cd905083b8-util" (OuterVolumeSpecName: "util") pod "6e054e24-b4cf-43ec-ba86-01cd905083b8" (UID: "6e054e24-b4cf-43ec-ba86-01cd905083b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:54:58.082205 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.082179 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef28f64b-9d52-441c-89b3-1a81294a172b-util\") pod \"ef28f64b-9d52-441c-89b3-1a81294a172b\" (UID: \"ef28f64b-9d52-441c-89b3-1a81294a172b\") " Sep 05 07:54:58.082320 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.082257 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8xm5\" (UniqueName: \"kubernetes.io/projected/ef28f64b-9d52-441c-89b3-1a81294a172b-kube-api-access-d8xm5\") pod \"ef28f64b-9d52-441c-89b3-1a81294a172b\" (UID: \"ef28f64b-9d52-441c-89b3-1a81294a172b\") " Sep 05 07:54:58.082320 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.082300 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef28f64b-9d52-441c-89b3-1a81294a172b-bundle\") pod \"ef28f64b-9d52-441c-89b3-1a81294a172b\" (UID: \"ef28f64b-9d52-441c-89b3-1a81294a172b\") " Sep 05 07:54:58.082476 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.082460 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2t9q\" (UniqueName: \"kubernetes.io/projected/6e054e24-b4cf-43ec-ba86-01cd905083b8-kube-api-access-n2t9q\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:54:58.082508 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.082482 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6e054e24-b4cf-43ec-ba86-01cd905083b8-util\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:54:58.082508 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.082497 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9be19cc8-3304-4522-a606-7652d9014932-util\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:54:58.082565 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.082509 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xxzc\" (UniqueName: \"kubernetes.io/projected/9be19cc8-3304-4522-a606-7652d9014932-kube-api-access-9xxzc\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:54:58.082565 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.082524 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6e054e24-b4cf-43ec-ba86-01cd905083b8-bundle\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:54:58.082565 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.082537 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9be19cc8-3304-4522-a606-7652d9014932-bundle\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:54:58.082756 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.082738 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef28f64b-9d52-441c-89b3-1a81294a172b-bundle" (OuterVolumeSpecName: "bundle") pod "ef28f64b-9d52-441c-89b3-1a81294a172b" (UID: "ef28f64b-9d52-441c-89b3-1a81294a172b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:54:58.084322 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.084301 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef28f64b-9d52-441c-89b3-1a81294a172b-kube-api-access-d8xm5" (OuterVolumeSpecName: "kube-api-access-d8xm5") pod "ef28f64b-9d52-441c-89b3-1a81294a172b" (UID: "ef28f64b-9d52-441c-89b3-1a81294a172b"). InnerVolumeSpecName "kube-api-access-d8xm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:54:58.087524 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.087505 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef28f64b-9d52-441c-89b3-1a81294a172b-util" (OuterVolumeSpecName: "util") pod "ef28f64b-9d52-441c-89b3-1a81294a172b" (UID: "ef28f64b-9d52-441c-89b3-1a81294a172b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:54:58.183282 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.183246 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ef28f64b-9d52-441c-89b3-1a81294a172b-bundle\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:54:58.183282 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.183281 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ef28f64b-9d52-441c-89b3-1a81294a172b-util\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:54:58.183422 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.183294 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8xm5\" (UniqueName: \"kubernetes.io/projected/ef28f64b-9d52-441c-89b3-1a81294a172b-kube-api-access-d8xm5\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:54:58.754700 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.754671 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" Sep 05 07:54:58.754889 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.754671 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0dpf6f" event={"ID":"ef28f64b-9d52-441c-89b3-1a81294a172b","Type":"ContainerDied","Data":"e6633cd062b53c53b077e00cc4a26fbf677ff36e8883bcfac40c7a1ad83f0d1d"} Sep 05 07:54:58.754889 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.754796 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6633cd062b53c53b077e00cc4a26fbf677ff36e8883bcfac40c7a1ad83f0d1d" Sep 05 07:54:58.756485 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.756467 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" Sep 05 07:54:58.756619 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.756492 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183cqhxt7" event={"ID":"9be19cc8-3304-4522-a606-7652d9014932","Type":"ContainerDied","Data":"29266f5a1d7ef1beada0046d375fced2a90d8273f5580c44fb09248bd6c07317"} Sep 05 07:54:58.756619 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.756525 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29266f5a1d7ef1beada0046d375fced2a90d8273f5580c44fb09248bd6c07317" Sep 05 07:54:58.758455 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.758437 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" Sep 05 07:54:58.758455 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.758443 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053vlcnn" event={"ID":"6e054e24-b4cf-43ec-ba86-01cd905083b8","Type":"ContainerDied","Data":"b72c5b864618c10cd267a059e4b5c933fa51d0adfd0f976358ab1eedcc9b7206"} Sep 05 07:54:58.758611 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.758479 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b72c5b864618c10cd267a059e4b5c933fa51d0adfd0f976358ab1eedcc9b7206" Sep 05 07:54:58.760609 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.760587 2576 generic.go:356] "Generic (PLEG): container finished" podID="ad6ae678-c862-41c4-b393-404f53fd97cd" containerID="7a872b765639ff83807ac29cef52121e949b853b0f3d4f076154e73c1fea33b4" exitCode=0 Sep 05 07:54:58.760700 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:58.760653 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" event={"ID":"ad6ae678-c862-41c4-b393-404f53fd97cd","Type":"ContainerDied","Data":"7a872b765639ff83807ac29cef52121e949b853b0f3d4f076154e73c1fea33b4"} Sep 05 07:54:59.767605 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:59.767567 2576 generic.go:356] "Generic (PLEG): container finished" podID="ad6ae678-c862-41c4-b393-404f53fd97cd" containerID="64ae01528d1878ce147b2515e756f903db7cf2da3b3f7e6f5b242dcc0f922b5c" exitCode=0 Sep 05 07:54:59.768042 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:54:59.767632 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" event={"ID":"ad6ae678-c862-41c4-b393-404f53fd97cd","Type":"ContainerDied","Data":"64ae01528d1878ce147b2515e756f903db7cf2da3b3f7e6f5b242dcc0f922b5c"} Sep 05 07:55:00.890388 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:00.890365 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" Sep 05 07:55:01.003922 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:01.003880 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad6ae678-c862-41c4-b393-404f53fd97cd-util\") pod \"ad6ae678-c862-41c4-b393-404f53fd97cd\" (UID: \"ad6ae678-c862-41c4-b393-404f53fd97cd\") " Sep 05 07:55:01.003922 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:01.003927 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24pd4\" (UniqueName: \"kubernetes.io/projected/ad6ae678-c862-41c4-b393-404f53fd97cd-kube-api-access-24pd4\") pod \"ad6ae678-c862-41c4-b393-404f53fd97cd\" (UID: \"ad6ae678-c862-41c4-b393-404f53fd97cd\") " Sep 05 07:55:01.004114 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:01.003981 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad6ae678-c862-41c4-b393-404f53fd97cd-bundle\") pod \"ad6ae678-c862-41c4-b393-404f53fd97cd\" (UID: \"ad6ae678-c862-41c4-b393-404f53fd97cd\") " Sep 05 07:55:01.004625 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:01.004600 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6ae678-c862-41c4-b393-404f53fd97cd-bundle" (OuterVolumeSpecName: "bundle") pod "ad6ae678-c862-41c4-b393-404f53fd97cd" (UID: "ad6ae678-c862-41c4-b393-404f53fd97cd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:55:01.006165 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:01.006137 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad6ae678-c862-41c4-b393-404f53fd97cd-kube-api-access-24pd4" (OuterVolumeSpecName: "kube-api-access-24pd4") pod "ad6ae678-c862-41c4-b393-404f53fd97cd" (UID: "ad6ae678-c862-41c4-b393-404f53fd97cd"). InnerVolumeSpecName "kube-api-access-24pd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:55:01.008484 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:01.008464 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad6ae678-c862-41c4-b393-404f53fd97cd-util" (OuterVolumeSpecName: "util") pod "ad6ae678-c862-41c4-b393-404f53fd97cd" (UID: "ad6ae678-c862-41c4-b393-404f53fd97cd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:55:01.105051 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:01.105014 2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad6ae678-c862-41c4-b393-404f53fd97cd-util\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:55:01.105051 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:01.105047 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24pd4\" (UniqueName: \"kubernetes.io/projected/ad6ae678-c862-41c4-b393-404f53fd97cd-kube-api-access-24pd4\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:55:01.105051 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:01.105056 2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad6ae678-c862-41c4-b393-404f53fd97cd-bundle\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:55:01.781332 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:01.781305 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" Sep 05 07:55:01.781504 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:01.781304 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb75b2l" event={"ID":"ad6ae678-c862-41c4-b393-404f53fd97cd","Type":"ContainerDied","Data":"83eaa56e87776180edb2b7390a2946ce322a0c962b7604c0e2761127fc066ceb"} Sep 05 07:55:01.781504 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:01.781459 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83eaa56e87776180edb2b7390a2946ce322a0c962b7604c0e2761127fc066ceb" Sep 05 07:55:06.642472 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642437 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-czbkm"] Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642722 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef28f64b-9d52-441c-89b3-1a81294a172b" containerName="pull" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642736 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef28f64b-9d52-441c-89b3-1a81294a172b" containerName="pull" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642765 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad6ae678-c862-41c4-b393-404f53fd97cd" containerName="util" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642770 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad6ae678-c862-41c4-b393-404f53fd97cd" containerName="util" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642778 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9be19cc8-3304-4522-a606-7652d9014932" containerName="pull" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642783 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9be19cc8-3304-4522-a606-7652d9014932" containerName="pull" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642790 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e054e24-b4cf-43ec-ba86-01cd905083b8" containerName="extract" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642798 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e054e24-b4cf-43ec-ba86-01cd905083b8" containerName="extract" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642807 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9be19cc8-3304-4522-a606-7652d9014932" containerName="extract" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642812 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9be19cc8-3304-4522-a606-7652d9014932" containerName="extract" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642817 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad6ae678-c862-41c4-b393-404f53fd97cd" containerName="extract" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642822 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad6ae678-c862-41c4-b393-404f53fd97cd" containerName="extract" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642830 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e054e24-b4cf-43ec-ba86-01cd905083b8" containerName="util" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642835 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e054e24-b4cf-43ec-ba86-01cd905083b8" containerName="util" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642840 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef28f64b-9d52-441c-89b3-1a81294a172b" containerName="extract" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642846 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef28f64b-9d52-441c-89b3-1a81294a172b" containerName="extract" Sep 05 07:55:06.642845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642854 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef28f64b-9d52-441c-89b3-1a81294a172b" containerName="util" Sep 05 07:55:06.643350 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642859 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef28f64b-9d52-441c-89b3-1a81294a172b" containerName="util" Sep 05 07:55:06.643350 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642868 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e054e24-b4cf-43ec-ba86-01cd905083b8" containerName="pull" Sep 05 07:55:06.643350 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642875 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e054e24-b4cf-43ec-ba86-01cd905083b8" containerName="pull" Sep 05 07:55:06.643350 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642885 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad6ae678-c862-41c4-b393-404f53fd97cd" containerName="pull" Sep 05 07:55:06.643350 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642892 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad6ae678-c862-41c4-b393-404f53fd97cd" containerName="pull" Sep 05 07:55:06.643350 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642899 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9be19cc8-3304-4522-a606-7652d9014932" containerName="util" Sep 05 07:55:06.643350 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642904 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9be19cc8-3304-4522-a606-7652d9014932" containerName="util" Sep 05 07:55:06.643350 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642949 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9be19cc8-3304-4522-a606-7652d9014932" containerName="extract" Sep 05 07:55:06.643350 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642958 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad6ae678-c862-41c4-b393-404f53fd97cd" containerName="extract" Sep 05 07:55:06.643350 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642972 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e054e24-b4cf-43ec-ba86-01cd905083b8" containerName="extract" Sep 05 07:55:06.643350 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.642978 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef28f64b-9d52-441c-89b3-1a81294a172b" containerName="extract" Sep 05 07:55:06.646629 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.646613 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-czbkm" Sep 05 07:55:06.648699 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.648674 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 05 07:55:06.648839 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.648791 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 05 07:55:06.648839 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.648796 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-5pwbq\"" Sep 05 07:55:06.657041 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.657019 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-czbkm"] Sep 05 07:55:06.752763 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.752729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbkxq\" (UniqueName: \"kubernetes.io/projected/cdb06401-71c2-4153-b4ac-18cd7bda3f4a-kube-api-access-cbkxq\") pod \"authorino-operator-68d88948d8-czbkm\" (UID: \"cdb06401-71c2-4153-b4ac-18cd7bda3f4a\") " pod="kuadrant-system/authorino-operator-68d88948d8-czbkm" Sep 05 07:55:06.853467 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.853431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbkxq\" (UniqueName: \"kubernetes.io/projected/cdb06401-71c2-4153-b4ac-18cd7bda3f4a-kube-api-access-cbkxq\") pod \"authorino-operator-68d88948d8-czbkm\" (UID: \"cdb06401-71c2-4153-b4ac-18cd7bda3f4a\") " pod="kuadrant-system/authorino-operator-68d88948d8-czbkm" Sep 05 07:55:06.861657 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.861634 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbkxq\" (UniqueName: \"kubernetes.io/projected/cdb06401-71c2-4153-b4ac-18cd7bda3f4a-kube-api-access-cbkxq\") pod \"authorino-operator-68d88948d8-czbkm\" (UID: \"cdb06401-71c2-4153-b4ac-18cd7bda3f4a\") " pod="kuadrant-system/authorino-operator-68d88948d8-czbkm" Sep 05 07:55:06.957188 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:06.957095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-czbkm" Sep 05 07:55:07.233042 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:07.233020 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-czbkm"] Sep 05 07:55:07.235628 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:55:07.235586 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdb06401_71c2_4153_b4ac_18cd7bda3f4a.slice/crio-a6c7496570f1c1cfad4b764409b85330c8e8e3b27c091d62ea2a4fbcf54ea2ce WatchSource:0}: Error finding container a6c7496570f1c1cfad4b764409b85330c8e8e3b27c091d62ea2a4fbcf54ea2ce: Status 404 returned error can't find the container with id a6c7496570f1c1cfad4b764409b85330c8e8e3b27c091d62ea2a4fbcf54ea2ce Sep 05 07:55:07.809658 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:07.809619 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-czbkm" event={"ID":"cdb06401-71c2-4153-b4ac-18cd7bda3f4a","Type":"ContainerStarted","Data":"a6c7496570f1c1cfad4b764409b85330c8e8e3b27c091d62ea2a4fbcf54ea2ce"} Sep 05 07:55:08.971871 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:08.971841 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-x262b"] Sep 05 07:55:08.974680 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:08.974664 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-x262b" Sep 05 07:55:08.977287 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:08.977266 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-ncd94\"" Sep 05 07:55:08.977695 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:08.977675 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 05 07:55:08.996648 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:08.996624 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-x262b"] Sep 05 07:55:09.070466 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:09.070436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlq8h\" (UniqueName: \"kubernetes.io/projected/5b7a0023-07e1-4c29-9919-9aed2462f386-kube-api-access-xlq8h\") pod \"dns-operator-controller-manager-5985564776-x262b\" (UID: \"5b7a0023-07e1-4c29-9919-9aed2462f386\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-x262b" Sep 05 07:55:09.171079 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:09.171047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlq8h\" (UniqueName: \"kubernetes.io/projected/5b7a0023-07e1-4c29-9919-9aed2462f386-kube-api-access-xlq8h\") pod \"dns-operator-controller-manager-5985564776-x262b\" (UID: \"5b7a0023-07e1-4c29-9919-9aed2462f386\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-x262b" Sep 05 07:55:09.184600 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:09.184575 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlq8h\" (UniqueName: \"kubernetes.io/projected/5b7a0023-07e1-4c29-9919-9aed2462f386-kube-api-access-xlq8h\") pod \"dns-operator-controller-manager-5985564776-x262b\" (UID: \"5b7a0023-07e1-4c29-9919-9aed2462f386\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-x262b" Sep 05 07:55:09.285026 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:09.284950 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-x262b" Sep 05 07:55:09.584580 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:09.584556 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-x262b"] Sep 05 07:55:09.586440 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:55:09.586414 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b7a0023_07e1_4c29_9919_9aed2462f386.slice/crio-88e327f0beb64a7aecc9ad69d2956aa0c3b11f13248078c251521d616db5956a WatchSource:0}: Error finding container 88e327f0beb64a7aecc9ad69d2956aa0c3b11f13248078c251521d616db5956a: Status 404 returned error can't find the container with id 88e327f0beb64a7aecc9ad69d2956aa0c3b11f13248078c251521d616db5956a Sep 05 07:55:09.819932 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:09.819896 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-x262b" event={"ID":"5b7a0023-07e1-4c29-9919-9aed2462f386","Type":"ContainerStarted","Data":"88e327f0beb64a7aecc9ad69d2956aa0c3b11f13248078c251521d616db5956a"} Sep 05 07:55:12.839262 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:12.839201 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-x262b" event={"ID":"5b7a0023-07e1-4c29-9919-9aed2462f386","Type":"ContainerStarted","Data":"efa6a1b6d9a0186676c3cac772cd1abfbe151253952796a5e43364f4159fb404"} Sep 05 07:55:12.839694 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:12.839341 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-x262b" Sep 05 07:55:12.858491 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:12.858447 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-x262b" podStartSLOduration=2.416850954 podStartE2EDuration="4.858435721s" totalImagesPullingTime="2.441584767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:55:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:55:12.858396681 +0000 UTC m=+415.467371817" watchObservedRunningTime="2026-09-05 07:55:12.858435721 +0000 UTC m=+415.467410826" Sep 05 07:55:15.256154 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:15.256124 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6"] Sep 05 07:55:15.259546 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:15.259515 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" Sep 05 07:55:15.270836 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:15.270808 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-d9xcp\"" Sep 05 07:55:15.286807 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:15.286745 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6"] Sep 05 07:55:15.420684 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:15.420652 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hk7h\" (UniqueName: \"kubernetes.io/projected/c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60-kube-api-access-6hk7h\") pod \"limitador-operator-controller-manager-6c549b6d85-nkpq6\" (UID: \"c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" Sep 05 07:55:15.521392 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:15.521317 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hk7h\" (UniqueName: \"kubernetes.io/projected/c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60-kube-api-access-6hk7h\") pod \"limitador-operator-controller-manager-6c549b6d85-nkpq6\" (UID: \"c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" Sep 05 07:55:15.532506 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:15.532480 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hk7h\" (UniqueName: \"kubernetes.io/projected/c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60-kube-api-access-6hk7h\") pod \"limitador-operator-controller-manager-6c549b6d85-nkpq6\" (UID: \"c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" Sep 05 07:55:15.570433 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:15.570409 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" Sep 05 07:55:15.842577 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:15.842394 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6"] Sep 05 07:55:15.845183 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:55:15.845153 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04f9e8e_8d45_4b0f_ae39_b0dcb07a4d60.slice/crio-2ed8aa605d53fb13bf45e24198af1de009b225171e82b776133d388542823e40 WatchSource:0}: Error finding container 2ed8aa605d53fb13bf45e24198af1de009b225171e82b776133d388542823e40: Status 404 returned error can't find the container with id 2ed8aa605d53fb13bf45e24198af1de009b225171e82b776133d388542823e40 Sep 05 07:55:15.854731 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:15.854699 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" event={"ID":"c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60","Type":"ContainerStarted","Data":"2ed8aa605d53fb13bf45e24198af1de009b225171e82b776133d388542823e40"} Sep 05 07:55:17.868393 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:17.868304 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" event={"ID":"c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60","Type":"ContainerStarted","Data":"9bce9182d854e969670d261bca32de4a3b6eddbee6a94c83b3dd6c99ca395c07"} Sep 05 07:55:17.868802 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:17.868460 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" Sep 05 07:55:17.885842 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:17.885791 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" podStartSLOduration=1.152233198 podStartE2EDuration="2.885773561s" totalImagesPullingTime="1.733540363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:55:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:55:17.883773717 +0000 UTC m=+420.492748842" watchObservedRunningTime="2026-09-05 07:55:17.885773561 +0000 UTC m=+420.494748686" Sep 05 07:55:18.874116 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:18.874081 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-czbkm" event={"ID":"cdb06401-71c2-4153-b4ac-18cd7bda3f4a","Type":"ContainerStarted","Data":"5e35521aabe5a59e5f7e3e2b80e46f16f6ce3a48b9ba19c468ef82f215024750"} Sep 05 07:55:18.874482 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:18.874274 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-czbkm" Sep 05 07:55:18.892813 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:18.892769 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-czbkm" podStartSLOduration=1.844789397 podStartE2EDuration="12.892757098s" totalImagesPullingTime="11.047967701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:55:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:55:18.890313676 +0000 UTC m=+421.499288799" watchObservedRunningTime="2026-09-05 07:55:18.892757098 +0000 UTC m=+421.501732221" Sep 05 07:55:20.493652 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.493616 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr"] Sep 05 07:55:20.496786 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.496769 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" Sep 05 07:55:20.498828 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.498806 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 05 07:55:20.498953 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.498833 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t92sv\"" Sep 05 07:55:20.498953 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.498844 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 05 07:55:20.505202 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.505178 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr"] Sep 05 07:55:20.661500 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.661459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2098fd7-17ed-4507-81c4-7c889eba0659-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vxbgr\" (UID: \"b2098fd7-17ed-4507-81c4-7c889eba0659\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" Sep 05 07:55:20.661500 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.661492 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2098fd7-17ed-4507-81c4-7c889eba0659-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-vxbgr\" (UID: \"b2098fd7-17ed-4507-81c4-7c889eba0659\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" Sep 05 07:55:20.661712 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.661518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvdv4\" (UniqueName: \"kubernetes.io/projected/b2098fd7-17ed-4507-81c4-7c889eba0659-kube-api-access-gvdv4\") pod \"kuadrant-console-plugin-598f86468b-vxbgr\" (UID: \"b2098fd7-17ed-4507-81c4-7c889eba0659\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" Sep 05 07:55:20.762872 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.762787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2098fd7-17ed-4507-81c4-7c889eba0659-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vxbgr\" (UID: \"b2098fd7-17ed-4507-81c4-7c889eba0659\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" Sep 05 07:55:20.762872 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.762827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2098fd7-17ed-4507-81c4-7c889eba0659-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-vxbgr\" (UID: \"b2098fd7-17ed-4507-81c4-7c889eba0659\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" Sep 05 07:55:20.763093 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:55:20.762927 2576 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 05 07:55:20.763093 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:55:20.762988 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2098fd7-17ed-4507-81c4-7c889eba0659-plugin-serving-cert podName:b2098fd7-17ed-4507-81c4-7c889eba0659 nodeName:}" failed. No retries permitted until 2026-09-05 07:55:21.26297089 +0000 UTC m=+423.871945994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b2098fd7-17ed-4507-81c4-7c889eba0659-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-vxbgr" (UID: "b2098fd7-17ed-4507-81c4-7c889eba0659") : secret "plugin-serving-cert" not found Sep 05 07:55:20.763093 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.763007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvdv4\" (UniqueName: \"kubernetes.io/projected/b2098fd7-17ed-4507-81c4-7c889eba0659-kube-api-access-gvdv4\") pod \"kuadrant-console-plugin-598f86468b-vxbgr\" (UID: \"b2098fd7-17ed-4507-81c4-7c889eba0659\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" Sep 05 07:55:20.763555 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.763524 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b2098fd7-17ed-4507-81c4-7c889eba0659-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-vxbgr\" (UID: \"b2098fd7-17ed-4507-81c4-7c889eba0659\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" Sep 05 07:55:20.773244 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:20.773188 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvdv4\" (UniqueName: \"kubernetes.io/projected/b2098fd7-17ed-4507-81c4-7c889eba0659-kube-api-access-gvdv4\") pod \"kuadrant-console-plugin-598f86468b-vxbgr\" (UID: \"b2098fd7-17ed-4507-81c4-7c889eba0659\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" Sep 05 07:55:21.268382 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:21.268341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2098fd7-17ed-4507-81c4-7c889eba0659-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vxbgr\" (UID: \"b2098fd7-17ed-4507-81c4-7c889eba0659\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" Sep 05 07:55:21.270858 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:21.270834 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b2098fd7-17ed-4507-81c4-7c889eba0659-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-vxbgr\" (UID: \"b2098fd7-17ed-4507-81c4-7c889eba0659\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" Sep 05 07:55:21.406307 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:21.406274 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" Sep 05 07:55:21.679452 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:21.679410 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr"] Sep 05 07:55:21.683804 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:55:21.683777 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2098fd7_17ed_4507_81c4_7c889eba0659.slice/crio-c4036daa8c0fabc89b85447f11050d0de6ebb3c90f3171cc89cf2ff999ac6a84 WatchSource:0}: Error finding container c4036daa8c0fabc89b85447f11050d0de6ebb3c90f3171cc89cf2ff999ac6a84: Status 404 returned error can't find the container with id c4036daa8c0fabc89b85447f11050d0de6ebb3c90f3171cc89cf2ff999ac6a84 Sep 05 07:55:21.892884 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:21.892846 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" event={"ID":"b2098fd7-17ed-4507-81c4-7c889eba0659","Type":"ContainerStarted","Data":"c4036daa8c0fabc89b85447f11050d0de6ebb3c90f3171cc89cf2ff999ac6a84"} Sep 05 07:55:23.846194 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:23.846160 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-x262b" Sep 05 07:55:26.921730 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:26.921693 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" event={"ID":"b2098fd7-17ed-4507-81c4-7c889eba0659","Type":"ContainerStarted","Data":"019bc689608f5d0b1a5b3ee807f4704085c750293cbbd3beb0fa35a08a597957"} Sep 05 07:55:26.940115 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:26.940062 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-vxbgr" podStartSLOduration=2.343049635 podStartE2EDuration="6.940043501s" totalImagesPullingTime="4.596993866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:55:26.939758209 +0000 UTC m=+429.548733331" watchObservedRunningTime="2026-09-05 07:55:26.940043501 +0000 UTC m=+429.549018625" Sep 05 07:55:28.877281 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:28.877241 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" Sep 05 07:55:29.885814 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:29.885783 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-czbkm" Sep 05 07:55:31.955743 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:31.955713 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29"] Sep 05 07:55:31.960523 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:31.960502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" Sep 05 07:55:31.964382 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:31.964355 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dr9bw\"" Sep 05 07:55:31.973383 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:31.973361 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6"] Sep 05 07:55:31.973559 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:31.973525 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" podUID="c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60" containerName="manager" containerID="cri-o://9bce9182d854e969670d261bca32de4a3b6eddbee6a94c83b3dd6c99ca395c07" gracePeriod=2 Sep 05 07:55:31.978656 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:31.978594 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29"] Sep 05 07:55:31.988666 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:31.988640 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6"] Sep 05 07:55:32.010191 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:55:32.010148 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-nkpq6\" is forbidden: User \"system:node:ip-10-0-143-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-184.ec2.internal' and this object" podUID="c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" Sep 05 07:55:32.053682 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.053639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5eca8c-9d87-4683-905a-2dd146d85e40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-2sr29\" (UID: \"9e5eca8c-9d87-4683-905a-2dd146d85e40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" Sep 05 07:55:32.053870 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.053695 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xhl2\" (UniqueName: \"kubernetes.io/projected/9e5eca8c-9d87-4683-905a-2dd146d85e40-kube-api-access-5xhl2\") pod \"kuadrant-operator-controller-manager-6bd956ccf-2sr29\" (UID: \"9e5eca8c-9d87-4683-905a-2dd146d85e40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" Sep 05 07:55:32.154858 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.154828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xhl2\" (UniqueName: \"kubernetes.io/projected/9e5eca8c-9d87-4683-905a-2dd146d85e40-kube-api-access-5xhl2\") pod \"kuadrant-operator-controller-manager-6bd956ccf-2sr29\" (UID: \"9e5eca8c-9d87-4683-905a-2dd146d85e40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" Sep 05 07:55:32.155006 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.154897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5eca8c-9d87-4683-905a-2dd146d85e40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-2sr29\" (UID: \"9e5eca8c-9d87-4683-905a-2dd146d85e40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" Sep 05 07:55:32.155284 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.155265 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5eca8c-9d87-4683-905a-2dd146d85e40-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-2sr29\" (UID: \"9e5eca8c-9d87-4683-905a-2dd146d85e40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" Sep 05 07:55:32.172336 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.172306 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xhl2\" (UniqueName: \"kubernetes.io/projected/9e5eca8c-9d87-4683-905a-2dd146d85e40-kube-api-access-5xhl2\") pod \"kuadrant-operator-controller-manager-6bd956ccf-2sr29\" (UID: \"9e5eca8c-9d87-4683-905a-2dd146d85e40\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" Sep 05 07:55:32.206620 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.206562 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" Sep 05 07:55:32.208600 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:55:32.208572 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-nkpq6\" is forbidden: User \"system:node:ip-10-0-143-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-184.ec2.internal' and this object" podUID="c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" Sep 05 07:55:32.255689 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.255663 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hk7h\" (UniqueName: \"kubernetes.io/projected/c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60-kube-api-access-6hk7h\") pod \"c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60\" (UID: \"c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60\") " Sep 05 07:55:32.257957 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.257933 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60-kube-api-access-6hk7h" (OuterVolumeSpecName: "kube-api-access-6hk7h") pod "c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60" (UID: "c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60"). InnerVolumeSpecName "kube-api-access-6hk7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:55:32.271120 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.271096 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" Sep 05 07:55:32.356889 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.356858 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hk7h\" (UniqueName: \"kubernetes.io/projected/c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60-kube-api-access-6hk7h\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:55:32.556307 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.556283 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29"] Sep 05 07:55:32.558650 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:55:32.558625 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5eca8c_9d87_4683_905a_2dd146d85e40.slice/crio-25ed3fe823967566bb85a55d68ca926d909ff68713939925c362a25a017d23bb WatchSource:0}: Error finding container 25ed3fe823967566bb85a55d68ca926d909ff68713939925c362a25a017d23bb: Status 404 returned error can't find the container with id 25ed3fe823967566bb85a55d68ca926d909ff68713939925c362a25a017d23bb Sep 05 07:55:32.951994 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.951954 2576 generic.go:356] "Generic (PLEG): container finished" podID="c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60" containerID="9bce9182d854e969670d261bca32de4a3b6eddbee6a94c83b3dd6c99ca395c07" exitCode=0 Sep 05 07:55:32.952205 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.952010 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" Sep 05 07:55:32.952205 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.952053 2576 scope.go:117] "RemoveContainer" containerID="9bce9182d854e969670d261bca32de4a3b6eddbee6a94c83b3dd6c99ca395c07" Sep 05 07:55:32.953691 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.953657 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" event={"ID":"9e5eca8c-9d87-4683-905a-2dd146d85e40","Type":"ContainerStarted","Data":"25ed3fe823967566bb85a55d68ca926d909ff68713939925c362a25a017d23bb"} Sep 05 07:55:32.954304 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:55:32.954277 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-nkpq6\" is forbidden: User \"system:node:ip-10-0-143-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-184.ec2.internal' and this object" podUID="c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" Sep 05 07:55:32.961167 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.961152 2576 scope.go:117] "RemoveContainer" containerID="9bce9182d854e969670d261bca32de4a3b6eddbee6a94c83b3dd6c99ca395c07" Sep 05 07:55:32.961470 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:55:32.961438 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bce9182d854e969670d261bca32de4a3b6eddbee6a94c83b3dd6c99ca395c07\": container with ID starting with 9bce9182d854e969670d261bca32de4a3b6eddbee6a94c83b3dd6c99ca395c07 not found: ID does not exist" containerID="9bce9182d854e969670d261bca32de4a3b6eddbee6a94c83b3dd6c99ca395c07" Sep 05 07:55:32.961470 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:32.961459 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bce9182d854e969670d261bca32de4a3b6eddbee6a94c83b3dd6c99ca395c07"} err="failed to get container status \"9bce9182d854e969670d261bca32de4a3b6eddbee6a94c83b3dd6c99ca395c07\": rpc error: code = NotFound desc = could not find container \"9bce9182d854e969670d261bca32de4a3b6eddbee6a94c83b3dd6c99ca395c07\": container with ID starting with 9bce9182d854e969670d261bca32de4a3b6eddbee6a94c83b3dd6c99ca395c07 not found: ID does not exist" Sep 05 07:55:32.962459 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:55:32.962437 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-nkpq6\" is forbidden: User \"system:node:ip-10-0-143-184.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-184.ec2.internal' and this object" podUID="c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-nkpq6" Sep 05 07:55:33.888124 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:33.888088 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60" path="/var/lib/kubelet/pods/c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60/volumes" Sep 05 07:55:36.978997 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:36.978959 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" event={"ID":"9e5eca8c-9d87-4683-905a-2dd146d85e40","Type":"ContainerStarted","Data":"61da7d6175a5f5a1076d9e3ebe51e5be8f3945dd594e7c8afb38e93b6501f82a"} Sep 05 07:55:36.979421 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:36.979036 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" Sep 05 07:55:37.020608 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:37.020547 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" podStartSLOduration=2.639830017 podStartE2EDuration="6.02052581s" totalImagesPullingTime="3.380695793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:55:37.019956805 +0000 UTC m=+439.628931929" watchObservedRunningTime="2026-09-05 07:55:37.02052581 +0000 UTC m=+439.629500937" Sep 05 07:55:47.986172 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:47.986091 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" Sep 05 07:55:48.231941 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.231908 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc"] Sep 05 07:55:48.232264 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.232251 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60" containerName="manager" Sep 05 07:55:48.232318 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.232266 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60" containerName="manager" Sep 05 07:55:48.232353 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.232331 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c04f9e8e-8d45-4b0f-ae39-b0dcb07a4d60" containerName="manager" Sep 05 07:55:48.236843 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.236787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" Sep 05 07:55:48.252932 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.252906 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc"] Sep 05 07:55:48.291326 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.291295 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znhjc\" (UniqueName: \"kubernetes.io/projected/386f9de5-277f-4be8-bb35-4506e4451a6b-kube-api-access-znhjc\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-28dlc\" (UID: \"386f9de5-277f-4be8-bb35-4506e4451a6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" Sep 05 07:55:48.291450 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.291409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/386f9de5-277f-4be8-bb35-4506e4451a6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-28dlc\" (UID: \"386f9de5-277f-4be8-bb35-4506e4451a6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" Sep 05 07:55:48.392701 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.392666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/386f9de5-277f-4be8-bb35-4506e4451a6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-28dlc\" (UID: \"386f9de5-277f-4be8-bb35-4506e4451a6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" Sep 05 07:55:48.392701 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.392702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znhjc\" (UniqueName: \"kubernetes.io/projected/386f9de5-277f-4be8-bb35-4506e4451a6b-kube-api-access-znhjc\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-28dlc\" (UID: \"386f9de5-277f-4be8-bb35-4506e4451a6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" Sep 05 07:55:48.393030 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.393010 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/386f9de5-277f-4be8-bb35-4506e4451a6b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-28dlc\" (UID: \"386f9de5-277f-4be8-bb35-4506e4451a6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" Sep 05 07:55:48.401101 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.401078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znhjc\" (UniqueName: \"kubernetes.io/projected/386f9de5-277f-4be8-bb35-4506e4451a6b-kube-api-access-znhjc\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-28dlc\" (UID: \"386f9de5-277f-4be8-bb35-4506e4451a6b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" Sep 05 07:55:48.546238 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.546138 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" Sep 05 07:55:48.824277 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:48.824243 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc"] Sep 05 07:55:48.828040 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:55:48.828012 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386f9de5_277f_4be8_bb35_4506e4451a6b.slice/crio-7262a46343318ea74dfc59acf3f9e5cfb71ea43c66b16646d9fc1824908b611b WatchSource:0}: Error finding container 7262a46343318ea74dfc59acf3f9e5cfb71ea43c66b16646d9fc1824908b611b: Status 404 returned error can't find the container with id 7262a46343318ea74dfc59acf3f9e5cfb71ea43c66b16646d9fc1824908b611b Sep 05 07:55:49.043102 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:49.043066 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" event={"ID":"386f9de5-277f-4be8-bb35-4506e4451a6b","Type":"ContainerStarted","Data":"5e90f0655e104b23b9b0794d0cd1156f2c072d1a38abc7b2a8313193ac5bac77"} Sep 05 07:55:49.043515 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:49.043119 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" event={"ID":"386f9de5-277f-4be8-bb35-4506e4451a6b","Type":"ContainerStarted","Data":"7262a46343318ea74dfc59acf3f9e5cfb71ea43c66b16646d9fc1824908b611b"} Sep 05 07:55:49.043515 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:49.043161 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" Sep 05 07:55:49.062357 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:55:49.062310 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" podStartSLOduration=1.06229504 podStartE2EDuration="1.06229504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:55:49.06011674 +0000 UTC m=+451.669091864" watchObservedRunningTime="2026-09-05 07:55:49.06229504 +0000 UTC m=+451.671270162" Sep 05 07:56:00.050660 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:00.050624 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" Sep 05 07:56:00.104859 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:00.104828 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29"] Sep 05 07:56:00.105111 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:00.105055 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" podUID="9e5eca8c-9d87-4683-905a-2dd146d85e40" containerName="manager" containerID="cri-o://61da7d6175a5f5a1076d9e3ebe51e5be8f3945dd594e7c8afb38e93b6501f82a" gracePeriod=10 Sep 05 07:56:00.349638 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:00.349617 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" Sep 05 07:56:00.497569 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:00.497538 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5eca8c-9d87-4683-905a-2dd146d85e40-extensions-socket-volume\") pod \"9e5eca8c-9d87-4683-905a-2dd146d85e40\" (UID: \"9e5eca8c-9d87-4683-905a-2dd146d85e40\") " Sep 05 07:56:00.497731 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:00.497602 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xhl2\" (UniqueName: \"kubernetes.io/projected/9e5eca8c-9d87-4683-905a-2dd146d85e40-kube-api-access-5xhl2\") pod \"9e5eca8c-9d87-4683-905a-2dd146d85e40\" (UID: \"9e5eca8c-9d87-4683-905a-2dd146d85e40\") " Sep 05 07:56:00.498004 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:00.497978 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5eca8c-9d87-4683-905a-2dd146d85e40-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9e5eca8c-9d87-4683-905a-2dd146d85e40" (UID: "9e5eca8c-9d87-4683-905a-2dd146d85e40"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 07:56:00.499813 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:00.499792 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5eca8c-9d87-4683-905a-2dd146d85e40-kube-api-access-5xhl2" (OuterVolumeSpecName: "kube-api-access-5xhl2") pod "9e5eca8c-9d87-4683-905a-2dd146d85e40" (UID: "9e5eca8c-9d87-4683-905a-2dd146d85e40"). InnerVolumeSpecName "kube-api-access-5xhl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 07:56:00.598525 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:00.598469 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9e5eca8c-9d87-4683-905a-2dd146d85e40-extensions-socket-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:56:00.598525 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:00.598490 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xhl2\" (UniqueName: \"kubernetes.io/projected/9e5eca8c-9d87-4683-905a-2dd146d85e40-kube-api-access-5xhl2\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 07:56:01.104703 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:01.104670 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e5eca8c-9d87-4683-905a-2dd146d85e40" containerID="61da7d6175a5f5a1076d9e3ebe51e5be8f3945dd594e7c8afb38e93b6501f82a" exitCode=0 Sep 05 07:56:01.105157 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:01.104713 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" Sep 05 07:56:01.105157 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:01.104755 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" event={"ID":"9e5eca8c-9d87-4683-905a-2dd146d85e40","Type":"ContainerDied","Data":"61da7d6175a5f5a1076d9e3ebe51e5be8f3945dd594e7c8afb38e93b6501f82a"} Sep 05 07:56:01.105157 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:01.104812 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29" event={"ID":"9e5eca8c-9d87-4683-905a-2dd146d85e40","Type":"ContainerDied","Data":"25ed3fe823967566bb85a55d68ca926d909ff68713939925c362a25a017d23bb"} Sep 05 07:56:01.105157 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:01.104846 2576 scope.go:117] "RemoveContainer" containerID="61da7d6175a5f5a1076d9e3ebe51e5be8f3945dd594e7c8afb38e93b6501f82a" Sep 05 07:56:01.114223 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:01.114195 2576 scope.go:117] "RemoveContainer" containerID="61da7d6175a5f5a1076d9e3ebe51e5be8f3945dd594e7c8afb38e93b6501f82a" Sep 05 07:56:01.114493 ip-10-0-143-184 kubenswrapper[2576]: E0905 07:56:01.114473 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61da7d6175a5f5a1076d9e3ebe51e5be8f3945dd594e7c8afb38e93b6501f82a\": container with ID starting with 61da7d6175a5f5a1076d9e3ebe51e5be8f3945dd594e7c8afb38e93b6501f82a not found: ID does not exist" containerID="61da7d6175a5f5a1076d9e3ebe51e5be8f3945dd594e7c8afb38e93b6501f82a" Sep 05 07:56:01.114565 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:01.114500 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61da7d6175a5f5a1076d9e3ebe51e5be8f3945dd594e7c8afb38e93b6501f82a"} err="failed to get container status \"61da7d6175a5f5a1076d9e3ebe51e5be8f3945dd594e7c8afb38e93b6501f82a\": rpc error: code = NotFound desc = could not find container \"61da7d6175a5f5a1076d9e3ebe51e5be8f3945dd594e7c8afb38e93b6501f82a\": container with ID starting with 61da7d6175a5f5a1076d9e3ebe51e5be8f3945dd594e7c8afb38e93b6501f82a not found: ID does not exist" Sep 05 07:56:01.125998 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:01.125968 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29"] Sep 05 07:56:01.131568 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:01.131539 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-2sr29"] Sep 05 07:56:01.886295 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:01.886264 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e5eca8c-9d87-4683-905a-2dd146d85e40" path="/var/lib/kubelet/pods/9e5eca8c-9d87-4683-905a-2dd146d85e40/volumes" Sep 05 07:56:19.757705 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:19.757673 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 07:56:19.758114 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:19.757959 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e5eca8c-9d87-4683-905a-2dd146d85e40" containerName="manager" Sep 05 07:56:19.758114 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:19.757970 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e5eca8c-9d87-4683-905a-2dd146d85e40" containerName="manager" Sep 05 07:56:19.758114 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:19.758035 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e5eca8c-9d87-4683-905a-2dd146d85e40" containerName="manager" Sep 05 07:56:19.762161 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:19.762142 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf" Sep 05 07:56:19.764653 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:19.764626 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 05 07:56:19.795768 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:19.795740 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 07:56:19.859200 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:19.859169 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 07:56:19.946385 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:19.946345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91e7d7f1-b79c-4df5-b679-0a2310994202-config-file\") pod \"limitador-limitador-85d9db4d5d-dx9qf\" (UID: \"91e7d7f1-b79c-4df5-b679-0a2310994202\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf" Sep 05 07:56:19.946574 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:19.946398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hcfg\" (UniqueName: \"kubernetes.io/projected/91e7d7f1-b79c-4df5-b679-0a2310994202-kube-api-access-6hcfg\") pod \"limitador-limitador-85d9db4d5d-dx9qf\" (UID: \"91e7d7f1-b79c-4df5-b679-0a2310994202\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf" Sep 05 07:56:20.047916 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:20.047824 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hcfg\" (UniqueName: \"kubernetes.io/projected/91e7d7f1-b79c-4df5-b679-0a2310994202-kube-api-access-6hcfg\") pod \"limitador-limitador-85d9db4d5d-dx9qf\" (UID: \"91e7d7f1-b79c-4df5-b679-0a2310994202\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf" Sep 05 07:56:20.048378 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:20.048352 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91e7d7f1-b79c-4df5-b679-0a2310994202-config-file\") pod \"limitador-limitador-85d9db4d5d-dx9qf\" (UID: \"91e7d7f1-b79c-4df5-b679-0a2310994202\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf" Sep 05 07:56:20.048929 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:20.048907 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/91e7d7f1-b79c-4df5-b679-0a2310994202-config-file\") pod \"limitador-limitador-85d9db4d5d-dx9qf\" (UID: \"91e7d7f1-b79c-4df5-b679-0a2310994202\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf" Sep 05 07:56:20.056399 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:20.056379 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hcfg\" (UniqueName: \"kubernetes.io/projected/91e7d7f1-b79c-4df5-b679-0a2310994202-kube-api-access-6hcfg\") pod \"limitador-limitador-85d9db4d5d-dx9qf\" (UID: \"91e7d7f1-b79c-4df5-b679-0a2310994202\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf" Sep 05 07:56:20.072205 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:20.072185 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf" Sep 05 07:56:20.345370 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:20.345348 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 07:56:20.347614 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:56:20.347580 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e7d7f1_b79c_4df5_b679_0a2310994202.slice/crio-5bc046e74e288348226f8a384e7a67c1ed489338726f8d15bd5c71009092b6db WatchSource:0}: Error finding container 5bc046e74e288348226f8a384e7a67c1ed489338726f8d15bd5c71009092b6db: Status 404 returned error can't find the container with id 5bc046e74e288348226f8a384e7a67c1ed489338726f8d15bd5c71009092b6db Sep 05 07:56:21.204727 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:21.204694 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf" event={"ID":"91e7d7f1-b79c-4df5-b679-0a2310994202","Type":"ContainerStarted","Data":"5bc046e74e288348226f8a384e7a67c1ed489338726f8d15bd5c71009092b6db"} Sep 05 07:56:22.211846 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:22.211806 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf" event={"ID":"91e7d7f1-b79c-4df5-b679-0a2310994202","Type":"ContainerStarted","Data":"62c00108748c3ccf4be10fdbb8df692ec60cc5b094703fc0915004c811a94966"} Sep 05 07:56:22.212280 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:22.211945 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf" Sep 05 07:56:22.227689 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:22.227634 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf" podStartSLOduration=2.399736407 podStartE2EDuration="3.227614191s" totalImagesPullingTime="827.877784ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:56:22.226494527 +0000 UTC m=+484.835469650" watchObservedRunningTime="2026-09-05 07:56:22.227614191 +0000 UTC m=+484.836589331" Sep 05 07:56:33.219298 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:33.219269 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf" Sep 05 07:56:49.826738 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:49.826693 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6dd96474c5-rwpb7"] Sep 05 07:56:49.830398 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:49.830378 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd96474c5-rwpb7" Sep 05 07:56:49.832890 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:49.832868 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 05 07:56:49.833006 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:49.832868 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-c2j9p\"" Sep 05 07:56:49.837491 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:49.837464 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd96474c5-rwpb7"] Sep 05 07:56:49.877990 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:49.877955 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgzrf\" (UniqueName: \"kubernetes.io/projected/334f4438-60a2-4b10-be24-6dceaf28f1f6-kube-api-access-vgzrf\") pod \"authorino-6dd96474c5-rwpb7\" (UID: \"334f4438-60a2-4b10-be24-6dceaf28f1f6\") " pod="kuadrant-system/authorino-6dd96474c5-rwpb7" Sep 05 07:56:49.878137 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:49.878029 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/334f4438-60a2-4b10-be24-6dceaf28f1f6-tls-cert\") pod \"authorino-6dd96474c5-rwpb7\" (UID: \"334f4438-60a2-4b10-be24-6dceaf28f1f6\") " pod="kuadrant-system/authorino-6dd96474c5-rwpb7" Sep 05 07:56:49.978559 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:49.978522 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgzrf\" (UniqueName: \"kubernetes.io/projected/334f4438-60a2-4b10-be24-6dceaf28f1f6-kube-api-access-vgzrf\") pod \"authorino-6dd96474c5-rwpb7\" (UID: \"334f4438-60a2-4b10-be24-6dceaf28f1f6\") " pod="kuadrant-system/authorino-6dd96474c5-rwpb7" Sep 05 07:56:49.978737 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:49.978588 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/334f4438-60a2-4b10-be24-6dceaf28f1f6-tls-cert\") pod \"authorino-6dd96474c5-rwpb7\" (UID: \"334f4438-60a2-4b10-be24-6dceaf28f1f6\") " pod="kuadrant-system/authorino-6dd96474c5-rwpb7" Sep 05 07:56:49.981110 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:49.981082 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/334f4438-60a2-4b10-be24-6dceaf28f1f6-tls-cert\") pod \"authorino-6dd96474c5-rwpb7\" (UID: \"334f4438-60a2-4b10-be24-6dceaf28f1f6\") " pod="kuadrant-system/authorino-6dd96474c5-rwpb7" Sep 05 07:56:49.985786 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:49.985763 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgzrf\" (UniqueName: \"kubernetes.io/projected/334f4438-60a2-4b10-be24-6dceaf28f1f6-kube-api-access-vgzrf\") pod \"authorino-6dd96474c5-rwpb7\" (UID: \"334f4438-60a2-4b10-be24-6dceaf28f1f6\") " pod="kuadrant-system/authorino-6dd96474c5-rwpb7" Sep 05 07:56:50.141458 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:50.141363 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd96474c5-rwpb7" Sep 05 07:56:50.415623 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:50.415591 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6dd96474c5-rwpb7"] Sep 05 07:56:50.417796 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:56:50.417768 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod334f4438_60a2_4b10_be24_6dceaf28f1f6.slice/crio-b014096615f164f8ce04929fc28cd285298e24bc6bcfd2cae72d7623fb76173b WatchSource:0}: Error finding container b014096615f164f8ce04929fc28cd285298e24bc6bcfd2cae72d7623fb76173b: Status 404 returned error can't find the container with id b014096615f164f8ce04929fc28cd285298e24bc6bcfd2cae72d7623fb76173b Sep 05 07:56:51.358368 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:51.358325 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd96474c5-rwpb7" event={"ID":"334f4438-60a2-4b10-be24-6dceaf28f1f6","Type":"ContainerStarted","Data":"b014096615f164f8ce04929fc28cd285298e24bc6bcfd2cae72d7623fb76173b"} Sep 05 07:56:52.365232 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:52.365180 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd96474c5-rwpb7" event={"ID":"334f4438-60a2-4b10-be24-6dceaf28f1f6","Type":"ContainerStarted","Data":"00d62bed8f41c2e48c068f14fa452b62c0a2222c354a4d32d02b87beb1c14782"} Sep 05 07:56:52.379100 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:56:52.379045 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6dd96474c5-rwpb7" podStartSLOduration=1.870523251 podStartE2EDuration="3.379022922s" totalImagesPullingTime="1.508499671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:56:52.37815458 +0000 UTC m=+514.987129715" watchObservedRunningTime="2026-09-05 07:56:52.379022922 +0000 UTC m=+514.987998048" Sep 05 07:57:36.245991 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:57:36.245904 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 07:57:56.384989 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:57:56.384951 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 07:58:07.805744 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:07.805709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 07:58:13.880225 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:13.880172 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b"] Sep 05 07:58:13.884138 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:13.884108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:13.886586 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:13.886557 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 05 07:58:13.887142 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:13.887122 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 05 07:58:13.887262 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:13.887164 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 05 07:58:13.887262 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:13.887244 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7nfpk\"" Sep 05 07:58:13.896483 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:13.896456 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b"] Sep 05 07:58:13.970543 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:13.970505 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87f228a1-6d98-4c4e-b59d-f83d249b50e8-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:13.970720 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:13.970561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87f228a1-6d98-4c4e-b59d-f83d249b50e8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:13.970720 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:13.970677 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87f228a1-6d98-4c4e-b59d-f83d249b50e8-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:13.970720 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:13.970713 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87f228a1-6d98-4c4e-b59d-f83d249b50e8-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:13.970845 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:13.970827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87f228a1-6d98-4c4e-b59d-f83d249b50e8-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:13.970885 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:13.970868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtzd\" (UniqueName: \"kubernetes.io/projected/87f228a1-6d98-4c4e-b59d-f83d249b50e8-kube-api-access-zrtzd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.071736 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.071695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87f228a1-6d98-4c4e-b59d-f83d249b50e8-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.071736 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.071746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87f228a1-6d98-4c4e-b59d-f83d249b50e8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.071995 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.071766 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87f228a1-6d98-4c4e-b59d-f83d249b50e8-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.071995 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.071794 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87f228a1-6d98-4c4e-b59d-f83d249b50e8-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.071995 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.071846 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87f228a1-6d98-4c4e-b59d-f83d249b50e8-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.071995 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.071870 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtzd\" (UniqueName: \"kubernetes.io/projected/87f228a1-6d98-4c4e-b59d-f83d249b50e8-kube-api-access-zrtzd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.072187 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.072092 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87f228a1-6d98-4c4e-b59d-f83d249b50e8-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.072187 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.072142 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87f228a1-6d98-4c4e-b59d-f83d249b50e8-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.072337 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.072205 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87f228a1-6d98-4c4e-b59d-f83d249b50e8-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.074160 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.074139 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87f228a1-6d98-4c4e-b59d-f83d249b50e8-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.074445 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.074429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87f228a1-6d98-4c4e-b59d-f83d249b50e8-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.079695 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.079674 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtzd\" (UniqueName: \"kubernetes.io/projected/87f228a1-6d98-4c4e-b59d-f83d249b50e8-kube-api-access-zrtzd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-b659b\" (UID: \"87f228a1-6d98-4c4e-b59d-f83d249b50e8\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.196609 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.196507 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:14.473599 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.473544 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b"] Sep 05 07:58:14.476050 ip-10-0-143-184 kubenswrapper[2576]: W0905 07:58:14.476009 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f228a1_6d98_4c4e_b59d_f83d249b50e8.slice/crio-e97d75e1bb001be3a2fa7dd33fd1b374283e076fe0934ff4526db7960b9022c4 WatchSource:0}: Error finding container e97d75e1bb001be3a2fa7dd33fd1b374283e076fe0934ff4526db7960b9022c4: Status 404 returned error can't find the container with id e97d75e1bb001be3a2fa7dd33fd1b374283e076fe0934ff4526db7960b9022c4 Sep 05 07:58:14.778677 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.778589 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" event={"ID":"87f228a1-6d98-4c4e-b59d-f83d249b50e8","Type":"ContainerStarted","Data":"e97d75e1bb001be3a2fa7dd33fd1b374283e076fe0934ff4526db7960b9022c4"} Sep 05 07:58:14.846498 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:14.846460 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 07:58:15.785743 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:15.785654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" event={"ID":"87f228a1-6d98-4c4e-b59d-f83d249b50e8","Type":"ContainerStarted","Data":"6a4573869070590d1746a82240c8f638d2e426e3a14655c4b3111acea0318e29"} Sep 05 07:58:15.804682 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:15.804617 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" podStartSLOduration=1.768032271 podStartE2EDuration="2.804600233s" totalImagesPullingTime="1.036567962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 07:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 07:58:15.802185344 +0000 UTC m=+598.411160469" watchObservedRunningTime="2026-09-05 07:58:15.804600233 +0000 UTC m=+598.413575368" Sep 05 07:58:17.856515 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:17.856488 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 07:58:17.857294 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:17.857269 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 07:58:17.865768 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:17.865742 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 07:58:17.866560 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:17.866537 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 07:58:23.053567 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:23.053531 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 07:58:24.197458 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:24.197419 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:24.197969 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:24.197492 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:24.210462 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:24.210430 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:24.844542 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:24.844510 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-b659b" Sep 05 07:58:34.656444 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:34.656413 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 07:58:43.447098 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:58:43.447002 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 07:59:01.455937 ip-10-0-143-184 kubenswrapper[2576]: I0905 07:59:01.455898 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:00:00.149225 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:00.149172 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29809920-zn8n8"] Sep 05 08:00:00.152675 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:00.152657 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809920-zn8n8" Sep 05 08:00:00.155025 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:00.154997 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpmmp\"" Sep 05 08:00:00.162454 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:00.162432 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809920-zn8n8" Sep 05 08:00:00.164672 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:00.164644 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29809920-zn8n8"] Sep 05 08:00:00.440618 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:00.440541 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29809920-zn8n8"] Sep 05 08:00:00.447414 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:00:00.447332 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c20fd9_95e6_46ad_93a4_5021d1d39caa.slice/crio-33b6dc8b897a6ee67584dc3b6c0ed2a025d63042b2849d1e4f05db05e7650da9 WatchSource:0}: Error finding container 33b6dc8b897a6ee67584dc3b6c0ed2a025d63042b2849d1e4f05db05e7650da9: Status 404 returned error can't find the container with id 33b6dc8b897a6ee67584dc3b6c0ed2a025d63042b2849d1e4f05db05e7650da9 Sep 05 08:00:00.449009 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:00.448989 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 08:00:01.315306 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:01.315272 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809920-zn8n8" event={"ID":"74c20fd9-95e6-46ad-93a4-5021d1d39caa","Type":"ContainerStarted","Data":"33b6dc8b897a6ee67584dc3b6c0ed2a025d63042b2849d1e4f05db05e7650da9"} Sep 05 08:00:03.328978 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:03.328932 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809920-zn8n8" event={"ID":"74c20fd9-95e6-46ad-93a4-5021d1d39caa","Type":"ContainerStarted","Data":"6f6f0f67b4526f7769439c1daffe393fda9a732c5bfc68dec5229dea5574a5f0"} Sep 05 08:00:03.345965 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:03.345910 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809920-zn8n8" podStartSLOduration=0.850159251 podStartE2EDuration="3.345892894s" totalImagesPullingTime="2.495733643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:00:03.343850969 +0000 UTC m=+705.952826104" watchObservedRunningTime="2026-09-05 08:00:03.345892894 +0000 UTC m=+705.954868017" Sep 05 08:00:04.335675 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:04.335634 2576 generic.go:356] "Generic (PLEG): container finished" podID="74c20fd9-95e6-46ad-93a4-5021d1d39caa" containerID="6f6f0f67b4526f7769439c1daffe393fda9a732c5bfc68dec5229dea5574a5f0" exitCode=0 Sep 05 08:00:04.336075 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:04.335725 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809920-zn8n8" event={"ID":"74c20fd9-95e6-46ad-93a4-5021d1d39caa","Type":"ContainerDied","Data":"6f6f0f67b4526f7769439c1daffe393fda9a732c5bfc68dec5229dea5574a5f0"} Sep 05 08:00:05.472000 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:05.471976 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809920-zn8n8" Sep 05 08:00:06.347653 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:06.347625 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809920-zn8n8" Sep 05 08:00:06.347834 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:06.347654 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809920-zn8n8" event={"ID":"74c20fd9-95e6-46ad-93a4-5021d1d39caa","Type":"ContainerDied","Data":"33b6dc8b897a6ee67584dc3b6c0ed2a025d63042b2849d1e4f05db05e7650da9"} Sep 05 08:00:06.347834 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:06.347697 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33b6dc8b897a6ee67584dc3b6c0ed2a025d63042b2849d1e4f05db05e7650da9" Sep 05 08:00:17.034052 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.033951 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd96474c5-rwpb7"] Sep 05 08:00:17.034560 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.034204 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6dd96474c5-rwpb7" podUID="334f4438-60a2-4b10-be24-6dceaf28f1f6" containerName="authorino" containerID="cri-o://00d62bed8f41c2e48c068f14fa452b62c0a2222c354a4d32d02b87beb1c14782" gracePeriod=30 Sep 05 08:00:17.287397 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.287336 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd96474c5-rwpb7" Sep 05 08:00:17.402672 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.402643 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgzrf\" (UniqueName: \"kubernetes.io/projected/334f4438-60a2-4b10-be24-6dceaf28f1f6-kube-api-access-vgzrf\") pod \"334f4438-60a2-4b10-be24-6dceaf28f1f6\" (UID: \"334f4438-60a2-4b10-be24-6dceaf28f1f6\") " Sep 05 08:00:17.402865 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.402725 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/334f4438-60a2-4b10-be24-6dceaf28f1f6-tls-cert\") pod \"334f4438-60a2-4b10-be24-6dceaf28f1f6\" (UID: \"334f4438-60a2-4b10-be24-6dceaf28f1f6\") " Sep 05 08:00:17.405048 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.405020 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334f4438-60a2-4b10-be24-6dceaf28f1f6-kube-api-access-vgzrf" (OuterVolumeSpecName: "kube-api-access-vgzrf") pod "334f4438-60a2-4b10-be24-6dceaf28f1f6" (UID: "334f4438-60a2-4b10-be24-6dceaf28f1f6"). InnerVolumeSpecName "kube-api-access-vgzrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:00:17.408151 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.408120 2576 generic.go:356] "Generic (PLEG): container finished" podID="334f4438-60a2-4b10-be24-6dceaf28f1f6" containerID="00d62bed8f41c2e48c068f14fa452b62c0a2222c354a4d32d02b87beb1c14782" exitCode=0 Sep 05 08:00:17.408295 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.408163 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6dd96474c5-rwpb7" Sep 05 08:00:17.408295 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.408179 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd96474c5-rwpb7" event={"ID":"334f4438-60a2-4b10-be24-6dceaf28f1f6","Type":"ContainerDied","Data":"00d62bed8f41c2e48c068f14fa452b62c0a2222c354a4d32d02b87beb1c14782"} Sep 05 08:00:17.408295 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.408230 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6dd96474c5-rwpb7" event={"ID":"334f4438-60a2-4b10-be24-6dceaf28f1f6","Type":"ContainerDied","Data":"b014096615f164f8ce04929fc28cd285298e24bc6bcfd2cae72d7623fb76173b"} Sep 05 08:00:17.408295 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.408253 2576 scope.go:117] "RemoveContainer" containerID="00d62bed8f41c2e48c068f14fa452b62c0a2222c354a4d32d02b87beb1c14782" Sep 05 08:00:17.413520 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.413498 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/334f4438-60a2-4b10-be24-6dceaf28f1f6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "334f4438-60a2-4b10-be24-6dceaf28f1f6" (UID: "334f4438-60a2-4b10-be24-6dceaf28f1f6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:00:17.421929 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.421916 2576 scope.go:117] "RemoveContainer" containerID="00d62bed8f41c2e48c068f14fa452b62c0a2222c354a4d32d02b87beb1c14782" Sep 05 08:00:17.422224 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:00:17.422193 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00d62bed8f41c2e48c068f14fa452b62c0a2222c354a4d32d02b87beb1c14782\": container with ID starting with 00d62bed8f41c2e48c068f14fa452b62c0a2222c354a4d32d02b87beb1c14782 not found: ID does not exist" containerID="00d62bed8f41c2e48c068f14fa452b62c0a2222c354a4d32d02b87beb1c14782" Sep 05 08:00:17.422294 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.422242 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00d62bed8f41c2e48c068f14fa452b62c0a2222c354a4d32d02b87beb1c14782"} err="failed to get container status \"00d62bed8f41c2e48c068f14fa452b62c0a2222c354a4d32d02b87beb1c14782\": rpc error: code = NotFound desc = could not find container \"00d62bed8f41c2e48c068f14fa452b62c0a2222c354a4d32d02b87beb1c14782\": container with ID starting with 00d62bed8f41c2e48c068f14fa452b62c0a2222c354a4d32d02b87beb1c14782 not found: ID does not exist" Sep 05 08:00:17.503859 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.503830 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgzrf\" (UniqueName: \"kubernetes.io/projected/334f4438-60a2-4b10-be24-6dceaf28f1f6-kube-api-access-vgzrf\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:00:17.503859 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.503861 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/334f4438-60a2-4b10-be24-6dceaf28f1f6-tls-cert\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:00:17.743025 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.742964 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6dd96474c5-rwpb7"] Sep 05 08:00:17.748854 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.748833 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6dd96474c5-rwpb7"] Sep 05 08:00:17.886294 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:17.886263 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="334f4438-60a2-4b10-be24-6dceaf28f1f6" path="/var/lib/kubelet/pods/334f4438-60a2-4b10-be24-6dceaf28f1f6/volumes" Sep 05 08:00:41.079193 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.079146 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7859-140262762760496"] Sep 05 08:00:41.079756 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.079741 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c20fd9-95e6-46ad-93a4-5021d1d39caa" containerName="cleanup" Sep 05 08:00:41.079756 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.079757 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c20fd9-95e6-46ad-93a4-5021d1d39caa" containerName="cleanup" Sep 05 08:00:41.079832 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.079767 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334f4438-60a2-4b10-be24-6dceaf28f1f6" containerName="authorino" Sep 05 08:00:41.079832 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.079772 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="334f4438-60a2-4b10-be24-6dceaf28f1f6" containerName="authorino" Sep 05 08:00:41.079896 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.079833 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c20fd9-95e6-46ad-93a4-5021d1d39caa" containerName="cleanup" Sep 05 08:00:41.079896 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.079844 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="334f4438-60a2-4b10-be24-6dceaf28f1f6" containerName="authorino" Sep 05 08:00:41.084750 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.084734 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7859-140262762760496" Sep 05 08:00:41.086065 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.086041 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn8dl\" (UniqueName: \"kubernetes.io/projected/ec39c682-02aa-4899-8bc7-c4dffc5735b9-kube-api-access-wn8dl\") pod \"test-curl-7859-140262762760496\" (UID: \"ec39c682-02aa-4899-8bc7-c4dffc5735b9\") " pod="opendatahub/test-curl-7859-140262762760496" Sep 05 08:00:41.090510 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.090491 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x4x7f\"" Sep 05 08:00:41.091060 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.091038 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7859-140262762760496"] Sep 05 08:00:41.186517 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.186486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn8dl\" (UniqueName: \"kubernetes.io/projected/ec39c682-02aa-4899-8bc7-c4dffc5735b9-kube-api-access-wn8dl\") pod \"test-curl-7859-140262762760496\" (UID: \"ec39c682-02aa-4899-8bc7-c4dffc5735b9\") " pod="opendatahub/test-curl-7859-140262762760496" Sep 05 08:00:41.195706 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.195685 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn8dl\" (UniqueName: \"kubernetes.io/projected/ec39c682-02aa-4899-8bc7-c4dffc5735b9-kube-api-access-wn8dl\") pod \"test-curl-7859-140262762760496\" (UID: \"ec39c682-02aa-4899-8bc7-c4dffc5735b9\") " pod="opendatahub/test-curl-7859-140262762760496" Sep 05 08:00:41.395343 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.395248 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7859-140262762760496" Sep 05 08:00:41.666415 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:41.666340 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7859-140262762760496"] Sep 05 08:00:41.669818 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:00:41.669784 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec39c682_02aa_4899_8bc7_c4dffc5735b9.slice/crio-3b76f3bd63341080b0979220d973619ce6a0b28a0becbc5d382942840c1b834f WatchSource:0}: Error finding container 3b76f3bd63341080b0979220d973619ce6a0b28a0becbc5d382942840c1b834f: Status 404 returned error can't find the container with id 3b76f3bd63341080b0979220d973619ce6a0b28a0becbc5d382942840c1b834f Sep 05 08:00:42.539164 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:42.539118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7859-140262762760496" event={"ID":"ec39c682-02aa-4899-8bc7-c4dffc5735b9","Type":"ContainerStarted","Data":"3b76f3bd63341080b0979220d973619ce6a0b28a0becbc5d382942840c1b834f"} Sep 05 08:00:42.862462 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:42.862431 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:00:43.365521 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.365486 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4"] Sep 05 08:00:43.374097 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.374073 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:43.377119 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.376997 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e7c0ff-serving-cert\"" Sep 05 08:00:43.382107 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.380151 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 08:00:43.382107 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.380179 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 05 08:00:43.382107 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.380236 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c62fz\"" Sep 05 08:00:43.382107 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.380603 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 05 08:00:43.382574 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.382549 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4"] Sep 05 08:00:43.402827 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.402804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pckk5\" (UniqueName: \"kubernetes.io/projected/17df67f4-2628-45ef-897c-0cb7345e11bc-kube-api-access-pckk5\") pod \"maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:43.402944 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.402835 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17df67f4-2628-45ef-897c-0cb7345e11bc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:43.402944 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.402867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:43.402944 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.402888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-metrics-certs\") pod \"maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:43.503403 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.503279 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckk5\" (UniqueName: \"kubernetes.io/projected/17df67f4-2628-45ef-897c-0cb7345e11bc-kube-api-access-pckk5\") pod \"maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:43.503403 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.503315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17df67f4-2628-45ef-897c-0cb7345e11bc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:43.503403 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.503341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:43.503403 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.503357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-metrics-certs\") pod \"maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:43.503650 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:00:43.503500 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-serving-cert: secret "maas-api-e2e-inf-a-e7c0ff-serving-cert" not found Sep 05 08:00:43.503650 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:00:43.503565 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-maas-api-tls podName:17df67f4-2628-45ef-897c-0cb7345e11bc nodeName:}" failed. No retries permitted until 2026-09-05 08:00:44.003545889 +0000 UTC m=+746.612521006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-maas-api-tls") pod "maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" (UID: "17df67f4-2628-45ef-897c-0cb7345e11bc") : secret "maas-api-e2e-inf-a-e7c0ff-serving-cert" not found Sep 05 08:00:43.504040 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.504013 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17df67f4-2628-45ef-897c-0cb7345e11bc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:43.506491 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.506468 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-metrics-certs\") pod \"maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:43.512561 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.512538 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pckk5\" (UniqueName: \"kubernetes.io/projected/17df67f4-2628-45ef-897c-0cb7345e11bc-kube-api-access-pckk5\") pod \"maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:43.546848 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.546815 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7859-140262762760496" event={"ID":"ec39c682-02aa-4899-8bc7-c4dffc5735b9","Type":"ContainerStarted","Data":"bd4c8bc8059b2e533d081d6f6a936c88c4eb69b3732848bdf62df401bc9fa013"} Sep 05 08:00:43.565641 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.565596 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7859-140262762760496" podStartSLOduration=0.8033484 podStartE2EDuration="2.565583167s" totalImagesPullingTime="1.762234767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:00:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:00:43.564249661 +0000 UTC m=+746.173224780" watchObservedRunningTime="2026-09-05 08:00:43.565583167 +0000 UTC m=+746.174558289" Sep 05 08:00:43.673454 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:43.673391 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7859-140262762760496_ec39c682-02aa-4899-8bc7-c4dffc5735b9/test-curl-7859-140262762760496/0.log" Sep 05 08:00:44.007631 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.007559 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:44.010161 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.010134 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-maas-api-tls\") pod \"maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:44.105120 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.105089 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5"] Sep 05 08:00:44.107287 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.107272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" Sep 05 08:00:44.107982 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.107959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c618f1d4-a0bd-4786-b3da-34b3db49ab5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5\" (UID: \"c618f1d4-a0bd-4786-b3da-34b3db49ab5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" Sep 05 08:00:44.108057 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.108000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjnsl\" (UniqueName: \"kubernetes.io/projected/c618f1d4-a0bd-4786-b3da-34b3db49ab5a-kube-api-access-mjnsl\") pod \"payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5\" (UID: \"c618f1d4-a0bd-4786-b3da-34b3db49ab5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" Sep 05 08:00:44.109846 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.109830 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-e7c0ff-dockercfg-mt7td\"" Sep 05 08:00:44.110243 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.110225 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-e7c0ff\"" Sep 05 08:00:44.122252 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.122200 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5"] Sep 05 08:00:44.127088 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.127069 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd"] Sep 05 08:00:44.135484 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.135465 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" Sep 05 08:00:44.145658 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.145634 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd"] Sep 05 08:00:44.208545 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.208507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c618f1d4-a0bd-4786-b3da-34b3db49ab5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5\" (UID: \"c618f1d4-a0bd-4786-b3da-34b3db49ab5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" Sep 05 08:00:44.208801 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.208554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjnsl\" (UniqueName: \"kubernetes.io/projected/c618f1d4-a0bd-4786-b3da-34b3db49ab5a-kube-api-access-mjnsl\") pod \"payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5\" (UID: \"c618f1d4-a0bd-4786-b3da-34b3db49ab5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" Sep 05 08:00:44.208801 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.208573 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9656cd-5bac-4230-8526-8c21a0a96d05-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd\" (UID: \"4b9656cd-5bac-4230-8526-8c21a0a96d05\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" Sep 05 08:00:44.208801 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.208588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m8qq\" (UniqueName: \"kubernetes.io/projected/4b9656cd-5bac-4230-8526-8c21a0a96d05-kube-api-access-2m8qq\") pod \"payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd\" (UID: \"4b9656cd-5bac-4230-8526-8c21a0a96d05\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" Sep 05 08:00:44.209444 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.209407 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c618f1d4-a0bd-4786-b3da-34b3db49ab5a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5\" (UID: \"c618f1d4-a0bd-4786-b3da-34b3db49ab5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" Sep 05 08:00:44.217391 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.217369 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjnsl\" (UniqueName: \"kubernetes.io/projected/c618f1d4-a0bd-4786-b3da-34b3db49ab5a-kube-api-access-mjnsl\") pod \"payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5\" (UID: \"c618f1d4-a0bd-4786-b3da-34b3db49ab5a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" Sep 05 08:00:44.291979 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.291909 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:44.309271 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.309238 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9656cd-5bac-4230-8526-8c21a0a96d05-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd\" (UID: \"4b9656cd-5bac-4230-8526-8c21a0a96d05\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" Sep 05 08:00:44.309387 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.309270 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m8qq\" (UniqueName: \"kubernetes.io/projected/4b9656cd-5bac-4230-8526-8c21a0a96d05-kube-api-access-2m8qq\") pod \"payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd\" (UID: \"4b9656cd-5bac-4230-8526-8c21a0a96d05\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" Sep 05 08:00:44.310181 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.310156 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9656cd-5bac-4230-8526-8c21a0a96d05-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd\" (UID: \"4b9656cd-5bac-4230-8526-8c21a0a96d05\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" Sep 05 08:00:44.317723 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.317699 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m8qq\" (UniqueName: \"kubernetes.io/projected/4b9656cd-5bac-4230-8526-8c21a0a96d05-kube-api-access-2m8qq\") pod \"payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd\" (UID: \"4b9656cd-5bac-4230-8526-8c21a0a96d05\") " pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" Sep 05 08:00:44.419077 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.419049 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" Sep 05 08:00:44.446804 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.446774 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" Sep 05 08:00:44.554975 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.554894 2576 generic.go:356] "Generic (PLEG): container finished" podID="ec39c682-02aa-4899-8bc7-c4dffc5735b9" containerID="bd4c8bc8059b2e533d081d6f6a936c88c4eb69b3732848bdf62df401bc9fa013" exitCode=0 Sep 05 08:00:44.554975 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.554958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7859-140262762760496" event={"ID":"ec39c682-02aa-4899-8bc7-c4dffc5735b9","Type":"ContainerDied","Data":"bd4c8bc8059b2e533d081d6f6a936c88c4eb69b3732848bdf62df401bc9fa013"} Sep 05 08:00:44.568766 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.568745 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4"] Sep 05 08:00:44.570268 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:00:44.570243 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17df67f4_2628_45ef_897c_0cb7345e11bc.slice/crio-e7b7c0f28edaadfb4d8e42f8860ebbcc10a4cad15b32c95666669d2ae4825b3c WatchSource:0}: Error finding container e7b7c0f28edaadfb4d8e42f8860ebbcc10a4cad15b32c95666669d2ae4825b3c: Status 404 returned error can't find the container with id e7b7c0f28edaadfb4d8e42f8860ebbcc10a4cad15b32c95666669d2ae4825b3c Sep 05 08:00:44.695658 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.695632 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5"] Sep 05 08:00:44.696986 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:00:44.696956 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc618f1d4_a0bd_4786_b3da_34b3db49ab5a.slice/crio-fdd72275dd5c038fee0f15935b17807ba73c00c26319261b2b680b8610931755 WatchSource:0}: Error finding container fdd72275dd5c038fee0f15935b17807ba73c00c26319261b2b680b8610931755: Status 404 returned error can't find the container with id fdd72275dd5c038fee0f15935b17807ba73c00c26319261b2b680b8610931755 Sep 05 08:00:44.736622 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:44.736601 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd"] Sep 05 08:00:44.738510 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:00:44.738481 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9656cd_5bac_4230_8526_8c21a0a96d05.slice/crio-480bcb710c38a8ef33b0f64fa95f552175c23252e8ba6cf6f774637389e8f916 WatchSource:0}: Error finding container 480bcb710c38a8ef33b0f64fa95f552175c23252e8ba6cf6f774637389e8f916: Status 404 returned error can't find the container with id 480bcb710c38a8ef33b0f64fa95f552175c23252e8ba6cf6f774637389e8f916 Sep 05 08:00:45.564865 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:45.564827 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" event={"ID":"4b9656cd-5bac-4230-8526-8c21a0a96d05","Type":"ContainerStarted","Data":"480bcb710c38a8ef33b0f64fa95f552175c23252e8ba6cf6f774637389e8f916"} Sep 05 08:00:45.566594 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:45.566528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" event={"ID":"c618f1d4-a0bd-4786-b3da-34b3db49ab5a","Type":"ContainerStarted","Data":"fdd72275dd5c038fee0f15935b17807ba73c00c26319261b2b680b8610931755"} Sep 05 08:00:45.568511 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:45.568484 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" event={"ID":"17df67f4-2628-45ef-897c-0cb7345e11bc","Type":"ContainerStarted","Data":"e7b7c0f28edaadfb4d8e42f8860ebbcc10a4cad15b32c95666669d2ae4825b3c"} Sep 05 08:00:45.766871 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:45.766839 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7859-140262762760496" Sep 05 08:00:45.800853 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:45.800820 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7859-140262762760496"] Sep 05 08:00:45.807927 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:45.807896 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7859-140262762760496"] Sep 05 08:00:45.823255 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:45.823171 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn8dl\" (UniqueName: \"kubernetes.io/projected/ec39c682-02aa-4899-8bc7-c4dffc5735b9-kube-api-access-wn8dl\") pod \"ec39c682-02aa-4899-8bc7-c4dffc5735b9\" (UID: \"ec39c682-02aa-4899-8bc7-c4dffc5735b9\") " Sep 05 08:00:45.826542 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:45.826517 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec39c682-02aa-4899-8bc7-c4dffc5735b9-kube-api-access-wn8dl" (OuterVolumeSpecName: "kube-api-access-wn8dl") pod "ec39c682-02aa-4899-8bc7-c4dffc5735b9" (UID: "ec39c682-02aa-4899-8bc7-c4dffc5735b9"). InnerVolumeSpecName "kube-api-access-wn8dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:00:45.889698 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:45.889666 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec39c682-02aa-4899-8bc7-c4dffc5735b9" path="/var/lib/kubelet/pods/ec39c682-02aa-4899-8bc7-c4dffc5735b9/volumes" Sep 05 08:00:45.924831 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:45.924784 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn8dl\" (UniqueName: \"kubernetes.io/projected/ec39c682-02aa-4899-8bc7-c4dffc5735b9-kube-api-access-wn8dl\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:00:46.576386 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:46.576348 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7859-140262762760496" Sep 05 08:00:46.576386 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:46.576378 2576 scope.go:117] "RemoveContainer" containerID="bd4c8bc8059b2e533d081d6f6a936c88c4eb69b3732848bdf62df401bc9fa013" Sep 05 08:00:47.583604 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:47.583571 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" event={"ID":"4b9656cd-5bac-4230-8526-8c21a0a96d05","Type":"ContainerStarted","Data":"a0a464d1cd6be280efaf26782fc9db11a4a4bcd7e6839ee6c81bf057b9d5d072"} Sep 05 08:00:47.584009 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:47.583638 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" Sep 05 08:00:47.585956 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:47.585932 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" event={"ID":"c618f1d4-a0bd-4786-b3da-34b3db49ab5a","Type":"ContainerStarted","Data":"31ac2c655a78ed0caa5cf2987348b98151799f7d9b3e25de304e3e682866dd7a"} Sep 05 08:00:47.586093 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:47.585992 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" Sep 05 08:00:47.587540 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:47.587512 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" event={"ID":"17df67f4-2628-45ef-897c-0cb7345e11bc","Type":"ContainerStarted","Data":"1f530e3717f7dfa918c6939a74b71819bdc7849e6e639add69cc2de9f42c3d5a"} Sep 05 08:00:47.587651 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:47.587637 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:47.625671 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:47.625623 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" podStartSLOduration=1.407285941 podStartE2EDuration="3.625605069s" totalImagesPullingTime="2.218319128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:00:47.622569242 +0000 UTC m=+750.231544367" watchObservedRunningTime="2026-09-05 08:00:47.625605069 +0000 UTC m=+750.234580195" Sep 05 08:00:47.660943 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:47.660901 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" podStartSLOduration=2.286427646 podStartE2EDuration="4.6608892s" totalImagesPullingTime="2.374461554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:00:47.657474159 +0000 UTC m=+750.266449282" watchObservedRunningTime="2026-09-05 08:00:47.6608892 +0000 UTC m=+750.269864322" Sep 05 08:00:47.688945 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:47.688899 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" podStartSLOduration=1.44159924 podStartE2EDuration="3.688881607s" totalImagesPullingTime="2.247282367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:00:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:00:47.685869247 +0000 UTC m=+750.294844369" watchObservedRunningTime="2026-09-05 08:00:47.688881607 +0000 UTC m=+750.297856728" Sep 05 08:00:52.527784 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.527749 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x"] Sep 05 08:00:52.528158 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.528092 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec39c682-02aa-4899-8bc7-c4dffc5735b9" containerName="test-curl-7859-140262762760496" Sep 05 08:00:52.528158 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.528103 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec39c682-02aa-4899-8bc7-c4dffc5735b9" containerName="test-curl-7859-140262762760496" Sep 05 08:00:52.528158 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.528156 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec39c682-02aa-4899-8bc7-c4dffc5735b9" containerName="test-curl-7859-140262762760496" Sep 05 08:00:52.530442 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.530420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" Sep 05 08:00:52.532873 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.532853 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-73e112cd\"" Sep 05 08:00:52.541369 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.541344 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x"] Sep 05 08:00:52.554014 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.553990 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74"] Sep 05 08:00:52.556387 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.556360 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" Sep 05 08:00:52.559849 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.559828 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-73e112cd-dockercfg-fps27\"" Sep 05 08:00:52.576522 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.576501 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74"] Sep 05 08:00:52.584486 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.584467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d106684-1c5e-4bab-822f-1d492dea0f31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x\" (UID: \"6d106684-1c5e-4bab-822f-1d492dea0f31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" Sep 05 08:00:52.584586 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.584496 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdtt5\" (UniqueName: \"kubernetes.io/projected/6d106684-1c5e-4bab-822f-1d492dea0f31-kube-api-access-sdtt5\") pod \"payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x\" (UID: \"6d106684-1c5e-4bab-822f-1d492dea0f31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" Sep 05 08:00:52.584586 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.584515 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c78217e-49fa-4d15-bec6-365ef8d35f4e-plugins-config-volume\") pod \"payload-processing-e2e-ait-73e112cd-7df759d875-8hw74\" (UID: \"7c78217e-49fa-4d15-bec6-365ef8d35f4e\") " pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" Sep 05 08:00:52.584704 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.584680 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp7l8\" (UniqueName: \"kubernetes.io/projected/7c78217e-49fa-4d15-bec6-365ef8d35f4e-kube-api-access-jp7l8\") pod \"payload-processing-e2e-ait-73e112cd-7df759d875-8hw74\" (UID: \"7c78217e-49fa-4d15-bec6-365ef8d35f4e\") " pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" Sep 05 08:00:52.685927 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.685883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp7l8\" (UniqueName: \"kubernetes.io/projected/7c78217e-49fa-4d15-bec6-365ef8d35f4e-kube-api-access-jp7l8\") pod \"payload-processing-e2e-ait-73e112cd-7df759d875-8hw74\" (UID: \"7c78217e-49fa-4d15-bec6-365ef8d35f4e\") " pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" Sep 05 08:00:52.686096 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.685939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d106684-1c5e-4bab-822f-1d492dea0f31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x\" (UID: \"6d106684-1c5e-4bab-822f-1d492dea0f31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" Sep 05 08:00:52.686096 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.685961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdtt5\" (UniqueName: \"kubernetes.io/projected/6d106684-1c5e-4bab-822f-1d492dea0f31-kube-api-access-sdtt5\") pod \"payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x\" (UID: \"6d106684-1c5e-4bab-822f-1d492dea0f31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" Sep 05 08:00:52.686096 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.685983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c78217e-49fa-4d15-bec6-365ef8d35f4e-plugins-config-volume\") pod \"payload-processing-e2e-ait-73e112cd-7df759d875-8hw74\" (UID: \"7c78217e-49fa-4d15-bec6-365ef8d35f4e\") " pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" Sep 05 08:00:52.686676 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.686656 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c78217e-49fa-4d15-bec6-365ef8d35f4e-plugins-config-volume\") pod \"payload-processing-e2e-ait-73e112cd-7df759d875-8hw74\" (UID: \"7c78217e-49fa-4d15-bec6-365ef8d35f4e\") " pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" Sep 05 08:00:52.686763 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.686745 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d106684-1c5e-4bab-822f-1d492dea0f31-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x\" (UID: \"6d106684-1c5e-4bab-822f-1d492dea0f31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" Sep 05 08:00:52.695483 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.695451 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdtt5\" (UniqueName: \"kubernetes.io/projected/6d106684-1c5e-4bab-822f-1d492dea0f31-kube-api-access-sdtt5\") pod \"payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x\" (UID: \"6d106684-1c5e-4bab-822f-1d492dea0f31\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" Sep 05 08:00:52.695789 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.695767 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp7l8\" (UniqueName: \"kubernetes.io/projected/7c78217e-49fa-4d15-bec6-365ef8d35f4e-kube-api-access-jp7l8\") pod \"payload-processing-e2e-ait-73e112cd-7df759d875-8hw74\" (UID: \"7c78217e-49fa-4d15-bec6-365ef8d35f4e\") " pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" Sep 05 08:00:52.840117 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.840082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" Sep 05 08:00:52.866679 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:52.866647 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" Sep 05 08:00:53.116912 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:53.116888 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x"] Sep 05 08:00:53.119154 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:00:53.119126 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d106684_1c5e_4bab_822f_1d492dea0f31.slice/crio-e81f30ddb2d9b86a6b471c77ee4e3e32dc8a3bc318ce50d81271db6c43a0019a WatchSource:0}: Error finding container e81f30ddb2d9b86a6b471c77ee4e3e32dc8a3bc318ce50d81271db6c43a0019a: Status 404 returned error can't find the container with id e81f30ddb2d9b86a6b471c77ee4e3e32dc8a3bc318ce50d81271db6c43a0019a Sep 05 08:00:53.153000 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:53.152971 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74"] Sep 05 08:00:53.154313 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:00:53.154290 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c78217e_49fa_4d15_bec6_365ef8d35f4e.slice/crio-9fb5bf539c735dae3e08f450ca96627062b4da9f17ffb7ac501a614a89fb1c14 WatchSource:0}: Error finding container 9fb5bf539c735dae3e08f450ca96627062b4da9f17ffb7ac501a614a89fb1c14: Status 404 returned error can't find the container with id 9fb5bf539c735dae3e08f450ca96627062b4da9f17ffb7ac501a614a89fb1c14 Sep 05 08:00:53.599855 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:53.599784 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:00:53.621961 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:53.621928 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" event={"ID":"7c78217e-49fa-4d15-bec6-365ef8d35f4e","Type":"ContainerStarted","Data":"c2de1559c1a0651fcc789a264d6f18933f63a1593f68f916db11ee86b7748105"} Sep 05 08:00:53.622145 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:53.621997 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" event={"ID":"7c78217e-49fa-4d15-bec6-365ef8d35f4e","Type":"ContainerStarted","Data":"9fb5bf539c735dae3e08f450ca96627062b4da9f17ffb7ac501a614a89fb1c14"} Sep 05 08:00:53.622145 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:53.622024 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" Sep 05 08:00:53.624009 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:53.623979 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" event={"ID":"6d106684-1c5e-4bab-822f-1d492dea0f31","Type":"ContainerStarted","Data":"fd0d0ce7ec450548896e7d23959caf855c7f504629f6cd18afb2f90bef045dce"} Sep 05 08:00:53.624152 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:53.624023 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" event={"ID":"6d106684-1c5e-4bab-822f-1d492dea0f31","Type":"ContainerStarted","Data":"e81f30ddb2d9b86a6b471c77ee4e3e32dc8a3bc318ce50d81271db6c43a0019a"} Sep 05 08:00:53.624152 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:53.624044 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" Sep 05 08:00:53.654003 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:53.653956 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" podStartSLOduration=1.653938983 podStartE2EDuration="1.653938983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:00:53.65307895 +0000 UTC m=+756.262054064" watchObservedRunningTime="2026-09-05 08:00:53.653938983 +0000 UTC m=+756.262914106" Sep 05 08:00:53.678447 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:53.678375 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" podStartSLOduration=1.6783551920000002 podStartE2EDuration="1.678355192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:00:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:00:53.677748534 +0000 UTC m=+756.286723657" watchObservedRunningTime="2026-09-05 08:00:53.678355192 +0000 UTC m=+756.287330315" Sep 05 08:00:56.170121 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.170051 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p"] Sep 05 08:00:56.172405 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.172390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" Sep 05 08:00:56.174816 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.174794 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-4faf97\"" Sep 05 08:00:56.174957 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.174911 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-4faf97-dockercfg-7tjwv\"" Sep 05 08:00:56.191333 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.191310 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p"] Sep 05 08:00:56.217544 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.217511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c-plugins-config-volume\") pod \"payload-processing-e2e-autores-4faf97-5c77597d99-df92p\" (UID: \"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c\") " pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" Sep 05 08:00:56.217670 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.217572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lklc\" (UniqueName: \"kubernetes.io/projected/d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c-kube-api-access-4lklc\") pod \"payload-processing-e2e-autores-4faf97-5c77597d99-df92p\" (UID: \"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c\") " pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" Sep 05 08:00:56.231935 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.231912 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p"] Sep 05 08:00:56.234119 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.234104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" Sep 05 08:00:56.250145 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.250122 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p"] Sep 05 08:00:56.288753 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.288715 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:00:56.318410 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.318374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2p7v\" (UniqueName: \"kubernetes.io/projected/22270ab3-14df-4da1-be90-ad5d48d59570-kube-api-access-c2p7v\") pod \"payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p\" (UID: \"22270ab3-14df-4da1-be90-ad5d48d59570\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" Sep 05 08:00:56.318634 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.318425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c-plugins-config-volume\") pod \"payload-processing-e2e-autores-4faf97-5c77597d99-df92p\" (UID: \"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c\") " pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" Sep 05 08:00:56.318634 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.318548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lklc\" (UniqueName: \"kubernetes.io/projected/d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c-kube-api-access-4lklc\") pod \"payload-processing-e2e-autores-4faf97-5c77597d99-df92p\" (UID: \"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c\") " pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" Sep 05 08:00:56.318634 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.318592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22270ab3-14df-4da1-be90-ad5d48d59570-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p\" (UID: \"22270ab3-14df-4da1-be90-ad5d48d59570\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" Sep 05 08:00:56.319232 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.319186 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c-plugins-config-volume\") pod \"payload-processing-e2e-autores-4faf97-5c77597d99-df92p\" (UID: \"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c\") " pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" Sep 05 08:00:56.329736 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.329703 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lklc\" (UniqueName: \"kubernetes.io/projected/d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c-kube-api-access-4lklc\") pod \"payload-processing-e2e-autores-4faf97-5c77597d99-df92p\" (UID: \"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c\") " pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" Sep 05 08:00:56.419832 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.419797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22270ab3-14df-4da1-be90-ad5d48d59570-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p\" (UID: \"22270ab3-14df-4da1-be90-ad5d48d59570\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" Sep 05 08:00:56.420001 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.419842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2p7v\" (UniqueName: \"kubernetes.io/projected/22270ab3-14df-4da1-be90-ad5d48d59570-kube-api-access-c2p7v\") pod \"payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p\" (UID: \"22270ab3-14df-4da1-be90-ad5d48d59570\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" Sep 05 08:00:56.420573 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.420526 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22270ab3-14df-4da1-be90-ad5d48d59570-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p\" (UID: \"22270ab3-14df-4da1-be90-ad5d48d59570\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" Sep 05 08:00:56.428247 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.428204 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2p7v\" (UniqueName: \"kubernetes.io/projected/22270ab3-14df-4da1-be90-ad5d48d59570-kube-api-access-c2p7v\") pod \"payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p\" (UID: \"22270ab3-14df-4da1-be90-ad5d48d59570\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" Sep 05 08:00:56.482871 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.482834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" Sep 05 08:00:56.543616 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.543573 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" Sep 05 08:00:56.768128 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.768105 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p"] Sep 05 08:00:56.770259 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:00:56.770232 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44d63b2_9f5f_4681_b7a6_0eb6e98c5f2c.slice/crio-9904e7caad563548383796c6d64bcffa4531e9bfd40b3681280e316620621cc7 WatchSource:0}: Error finding container 9904e7caad563548383796c6d64bcffa4531e9bfd40b3681280e316620621cc7: Status 404 returned error can't find the container with id 9904e7caad563548383796c6d64bcffa4531e9bfd40b3681280e316620621cc7 Sep 05 08:00:56.834572 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:56.834546 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p"] Sep 05 08:00:56.835724 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:00:56.835696 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22270ab3_14df_4da1_be90_ad5d48d59570.slice/crio-4bba1fa83eda9d763b64c8645a1fab2939dd69d495afabdb7af0d361611836a4 WatchSource:0}: Error finding container 4bba1fa83eda9d763b64c8645a1fab2939dd69d495afabdb7af0d361611836a4: Status 404 returned error can't find the container with id 4bba1fa83eda9d763b64c8645a1fab2939dd69d495afabdb7af0d361611836a4 Sep 05 08:00:57.652143 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:57.652103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" event={"ID":"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c","Type":"ContainerStarted","Data":"b417b4be96cc5e3064f59b19d1ac9a4faf32d6bd879210f094fd0f67fcb7862f"} Sep 05 08:00:57.652666 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:57.652172 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" event={"ID":"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c","Type":"ContainerStarted","Data":"9904e7caad563548383796c6d64bcffa4531e9bfd40b3681280e316620621cc7"} Sep 05 08:00:57.652666 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:57.652312 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" Sep 05 08:00:57.653836 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:57.653804 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" event={"ID":"22270ab3-14df-4da1-be90-ad5d48d59570","Type":"ContainerStarted","Data":"c4a77482001ef1c84e3333ed27be1b84ef8e01626bc45887d7a4de3fde1db3aa"} Sep 05 08:00:57.653909 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:57.653844 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" Sep 05 08:00:57.653909 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:57.653862 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" event={"ID":"22270ab3-14df-4da1-be90-ad5d48d59570","Type":"ContainerStarted","Data":"4bba1fa83eda9d763b64c8645a1fab2939dd69d495afabdb7af0d361611836a4"} Sep 05 08:00:57.672076 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:57.672023 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" podStartSLOduration=1.672004687 podStartE2EDuration="1.672004687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:00:57.669658112 +0000 UTC m=+760.278633234" watchObservedRunningTime="2026-09-05 08:00:57.672004687 +0000 UTC m=+760.280979812" Sep 05 08:00:57.690292 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:57.690239 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" podStartSLOduration=1.690201884 podStartE2EDuration="1.690201884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:00:57.686868524 +0000 UTC m=+760.295843647" watchObservedRunningTime="2026-09-05 08:00:57.690201884 +0000 UTC m=+760.299177009" Sep 05 08:00:58.595434 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:58.595399 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" Sep 05 08:00:58.595851 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:00:58.595828 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" Sep 05 08:01:04.631314 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:04.631274 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" Sep 05 08:01:04.631720 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:04.631456 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" Sep 05 08:01:08.661727 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:08.661696 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" Sep 05 08:01:08.662111 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:08.661771 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" Sep 05 08:01:11.934289 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:11.934252 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:01:12.315106 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.315079 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6"] Sep 05 08:01:12.326060 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.326032 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.331267 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.331242 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-61f3ac-serving-cert\"" Sep 05 08:01:12.335044 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.334549 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6"] Sep 05 08:01:12.469241 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.469170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6kg7\" (UniqueName: \"kubernetes.io/projected/6c9a112a-7ec9-48d6-869a-f51636111f78-kube-api-access-m6kg7\") pod \"maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.469422 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.469270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9a112a-7ec9-48d6-869a-f51636111f78-metrics-certs\") pod \"maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.469422 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.469309 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9a112a-7ec9-48d6-869a-f51636111f78-maas-api-tls\") pod \"maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.469422 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.469338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a112a-7ec9-48d6-869a-f51636111f78-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.570520 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.570434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6kg7\" (UniqueName: \"kubernetes.io/projected/6c9a112a-7ec9-48d6-869a-f51636111f78-kube-api-access-m6kg7\") pod \"maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.570520 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.570493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9a112a-7ec9-48d6-869a-f51636111f78-metrics-certs\") pod \"maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.570727 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.570532 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9a112a-7ec9-48d6-869a-f51636111f78-maas-api-tls\") pod \"maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.570727 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.570564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a112a-7ec9-48d6-869a-f51636111f78-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.571242 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.571202 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a112a-7ec9-48d6-869a-f51636111f78-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.573191 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.573168 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9a112a-7ec9-48d6-869a-f51636111f78-maas-api-tls\") pod \"maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.573321 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.573306 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9a112a-7ec9-48d6-869a-f51636111f78-metrics-certs\") pod \"maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.584447 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.584422 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6kg7\" (UniqueName: \"kubernetes.io/projected/6c9a112a-7ec9-48d6-869a-f51636111f78-kube-api-access-m6kg7\") pod \"maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.640305 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.640276 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:12.932484 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:12.932450 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6"] Sep 05 08:01:12.934752 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:01:12.934716 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9a112a_7ec9_48d6_869a_f51636111f78.slice/crio-18fb4cf22a54ab34331795ef031c4e5c32b0069aae7bd1257048dc7b1bdc9be9 WatchSource:0}: Error finding container 18fb4cf22a54ab34331795ef031c4e5c32b0069aae7bd1257048dc7b1bdc9be9: Status 404 returned error can't find the container with id 18fb4cf22a54ab34331795ef031c4e5c32b0069aae7bd1257048dc7b1bdc9be9 Sep 05 08:01:13.300323 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.300235 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx"] Sep 05 08:01:13.305956 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.305925 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" Sep 05 08:01:13.308540 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.308516 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-61f3ac\"" Sep 05 08:01:13.308677 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.308617 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-61f3ac-dockercfg-pbzqn\"" Sep 05 08:01:13.321297 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.321269 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx"] Sep 05 08:01:13.356460 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.356430 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv"] Sep 05 08:01:13.360618 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.360600 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" Sep 05 08:01:13.376189 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.376168 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv"] Sep 05 08:01:13.378711 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.378690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zr7z\" (UniqueName: \"kubernetes.io/projected/276bcafa-917b-4f8f-941b-0ad4080811c9-kube-api-access-6zr7z\") pod \"payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx\" (UID: \"276bcafa-917b-4f8f-941b-0ad4080811c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" Sep 05 08:01:13.378843 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.378732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276bcafa-917b-4f8f-941b-0ad4080811c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx\" (UID: \"276bcafa-917b-4f8f-941b-0ad4080811c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" Sep 05 08:01:13.479812 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.479773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zr7z\" (UniqueName: \"kubernetes.io/projected/276bcafa-917b-4f8f-941b-0ad4080811c9-kube-api-access-6zr7z\") pod \"payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx\" (UID: \"276bcafa-917b-4f8f-941b-0ad4080811c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" Sep 05 08:01:13.480006 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.479832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276bcafa-917b-4f8f-941b-0ad4080811c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx\" (UID: \"276bcafa-917b-4f8f-941b-0ad4080811c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" Sep 05 08:01:13.480006 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.479855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5ade3f-cf42-4f42-9607-e315ee08c76d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv\" (UID: \"0d5ade3f-cf42-4f42-9607-e315ee08c76d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" Sep 05 08:01:13.480006 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.479896 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74z4z\" (UniqueName: \"kubernetes.io/projected/0d5ade3f-cf42-4f42-9607-e315ee08c76d-kube-api-access-74z4z\") pod \"payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv\" (UID: \"0d5ade3f-cf42-4f42-9607-e315ee08c76d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" Sep 05 08:01:13.480643 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.480615 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276bcafa-917b-4f8f-941b-0ad4080811c9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx\" (UID: \"276bcafa-917b-4f8f-941b-0ad4080811c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" Sep 05 08:01:13.491729 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.491694 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zr7z\" (UniqueName: \"kubernetes.io/projected/276bcafa-917b-4f8f-941b-0ad4080811c9-kube-api-access-6zr7z\") pod \"payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx\" (UID: \"276bcafa-917b-4f8f-941b-0ad4080811c9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" Sep 05 08:01:13.581017 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.580978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5ade3f-cf42-4f42-9607-e315ee08c76d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv\" (UID: \"0d5ade3f-cf42-4f42-9607-e315ee08c76d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" Sep 05 08:01:13.581206 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.581056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74z4z\" (UniqueName: \"kubernetes.io/projected/0d5ade3f-cf42-4f42-9607-e315ee08c76d-kube-api-access-74z4z\") pod \"payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv\" (UID: \"0d5ade3f-cf42-4f42-9607-e315ee08c76d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" Sep 05 08:01:13.581829 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.581805 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5ade3f-cf42-4f42-9607-e315ee08c76d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv\" (UID: \"0d5ade3f-cf42-4f42-9607-e315ee08c76d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" Sep 05 08:01:13.593038 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.593009 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74z4z\" (UniqueName: \"kubernetes.io/projected/0d5ade3f-cf42-4f42-9607-e315ee08c76d-kube-api-access-74z4z\") pod \"payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv\" (UID: \"0d5ade3f-cf42-4f42-9607-e315ee08c76d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" Sep 05 08:01:13.616934 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.616908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" Sep 05 08:01:13.671793 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.671754 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" Sep 05 08:01:13.751985 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.751943 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" event={"ID":"6c9a112a-7ec9-48d6-869a-f51636111f78","Type":"ContainerStarted","Data":"cdcc027dc1feef36a3d143261b41bb5ee6fcc9dd252a2276bffb02f991584f8d"} Sep 05 08:01:13.752138 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.752005 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" event={"ID":"6c9a112a-7ec9-48d6-869a-f51636111f78","Type":"ContainerStarted","Data":"18fb4cf22a54ab34331795ef031c4e5c32b0069aae7bd1257048dc7b1bdc9be9"} Sep 05 08:01:13.752138 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.752065 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:13.773902 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.773853 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" podStartSLOduration=1.3424925079999999 podStartE2EDuration="1.773837224s" totalImagesPullingTime="431.344716ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:01:13.772603961 +0000 UTC m=+776.381579098" watchObservedRunningTime="2026-09-05 08:01:13.773837224 +0000 UTC m=+776.382812405" Sep 05 08:01:13.903656 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.903628 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx"] Sep 05 08:01:13.904718 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:01:13.904694 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276bcafa_917b_4f8f_941b_0ad4080811c9.slice/crio-26508abe15817c8713113f308438887edcd716f592516a4d6def2162b27a2770 WatchSource:0}: Error finding container 26508abe15817c8713113f308438887edcd716f592516a4d6def2162b27a2770: Status 404 returned error can't find the container with id 26508abe15817c8713113f308438887edcd716f592516a4d6def2162b27a2770 Sep 05 08:01:13.957115 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:13.957092 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv"] Sep 05 08:01:13.958862 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:01:13.958835 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d5ade3f_cf42_4f42_9607_e315ee08c76d.slice/crio-bc1afc52f88605b509f7422c4cf2877118897a5f81566660d22e647f6a6f415c WatchSource:0}: Error finding container bc1afc52f88605b509f7422c4cf2877118897a5f81566660d22e647f6a6f415c: Status 404 returned error can't find the container with id bc1afc52f88605b509f7422c4cf2877118897a5f81566660d22e647f6a6f415c Sep 05 08:01:14.762602 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:14.762566 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" event={"ID":"0d5ade3f-cf42-4f42-9607-e315ee08c76d","Type":"ContainerStarted","Data":"3b192f9c50fcd45dfc3f8e6629435827306255db7b1991d18c412610c0b659ef"} Sep 05 08:01:14.762808 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:14.762620 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" event={"ID":"0d5ade3f-cf42-4f42-9607-e315ee08c76d","Type":"ContainerStarted","Data":"bc1afc52f88605b509f7422c4cf2877118897a5f81566660d22e647f6a6f415c"} Sep 05 08:01:14.762808 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:14.762680 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" Sep 05 08:01:14.764447 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:14.764421 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" event={"ID":"276bcafa-917b-4f8f-941b-0ad4080811c9","Type":"ContainerStarted","Data":"97a16875192e9ff69439380c475181d20108fcc51c40e34310cb5ffcdf777f32"} Sep 05 08:01:14.764569 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:14.764462 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" event={"ID":"276bcafa-917b-4f8f-941b-0ad4080811c9","Type":"ContainerStarted","Data":"26508abe15817c8713113f308438887edcd716f592516a4d6def2162b27a2770"} Sep 05 08:01:14.764569 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:14.764564 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" Sep 05 08:01:14.784205 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:14.784143 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" podStartSLOduration=1.7841310620000002 podStartE2EDuration="1.784131062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:01:14.782351572 +0000 UTC m=+777.391326697" watchObservedRunningTime="2026-09-05 08:01:14.784131062 +0000 UTC m=+777.393106185" Sep 05 08:01:14.815038 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:14.814981 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" podStartSLOduration=1.814960978 podStartE2EDuration="1.814960978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:01:14.813282257 +0000 UTC m=+777.422257381" watchObservedRunningTime="2026-09-05 08:01:14.814960978 +0000 UTC m=+777.423936104" Sep 05 08:01:19.770301 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:19.770271 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:01:25.771928 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:25.771896 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" Sep 05 08:01:25.772424 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:25.771977 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" Sep 05 08:01:28.355091 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:28.355056 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:01:32.861047 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:32.861013 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74"] Sep 05 08:01:32.861556 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:32.861278 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" podUID="7c78217e-49fa-4d15-bec6-365ef8d35f4e" containerName="payload-processing" containerID="cri-o://c2de1559c1a0651fcc789a264d6f18933f63a1593f68f916db11ee86b7748105" gracePeriod=30 Sep 05 08:01:32.875615 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:32.875586 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x"] Sep 05 08:01:32.875851 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:32.875824 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" podUID="6d106684-1c5e-4bab-822f-1d492dea0f31" containerName="payload-pre-processing" containerID="cri-o://fd0d0ce7ec450548896e7d23959caf855c7f504629f6cd18afb2f90bef045dce" gracePeriod=30 Sep 05 08:01:33.120566 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.120505 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" Sep 05 08:01:33.150976 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.150955 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" Sep 05 08:01:33.159332 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.159303 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c78217e-49fa-4d15-bec6-365ef8d35f4e-plugins-config-volume\") pod \"7c78217e-49fa-4d15-bec6-365ef8d35f4e\" (UID: \"7c78217e-49fa-4d15-bec6-365ef8d35f4e\") " Sep 05 08:01:33.159453 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.159422 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp7l8\" (UniqueName: \"kubernetes.io/projected/7c78217e-49fa-4d15-bec6-365ef8d35f4e-kube-api-access-jp7l8\") pod \"7c78217e-49fa-4d15-bec6-365ef8d35f4e\" (UID: \"7c78217e-49fa-4d15-bec6-365ef8d35f4e\") " Sep 05 08:01:33.159677 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.159650 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c78217e-49fa-4d15-bec6-365ef8d35f4e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7c78217e-49fa-4d15-bec6-365ef8d35f4e" (UID: "7c78217e-49fa-4d15-bec6-365ef8d35f4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:01:33.159885 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.159866 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c78217e-49fa-4d15-bec6-365ef8d35f4e-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:01:33.161546 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.161522 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c78217e-49fa-4d15-bec6-365ef8d35f4e-kube-api-access-jp7l8" (OuterVolumeSpecName: "kube-api-access-jp7l8") pod "7c78217e-49fa-4d15-bec6-365ef8d35f4e" (UID: "7c78217e-49fa-4d15-bec6-365ef8d35f4e"). InnerVolumeSpecName "kube-api-access-jp7l8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:01:33.260296 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.260268 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdtt5\" (UniqueName: \"kubernetes.io/projected/6d106684-1c5e-4bab-822f-1d492dea0f31-kube-api-access-sdtt5\") pod \"6d106684-1c5e-4bab-822f-1d492dea0f31\" (UID: \"6d106684-1c5e-4bab-822f-1d492dea0f31\") " Sep 05 08:01:33.260469 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.260380 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d106684-1c5e-4bab-822f-1d492dea0f31-plugins-config-volume\") pod \"6d106684-1c5e-4bab-822f-1d492dea0f31\" (UID: \"6d106684-1c5e-4bab-822f-1d492dea0f31\") " Sep 05 08:01:33.260561 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.260543 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp7l8\" (UniqueName: \"kubernetes.io/projected/7c78217e-49fa-4d15-bec6-365ef8d35f4e-kube-api-access-jp7l8\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:01:33.260744 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.260726 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d106684-1c5e-4bab-822f-1d492dea0f31-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d106684-1c5e-4bab-822f-1d492dea0f31" (UID: "6d106684-1c5e-4bab-822f-1d492dea0f31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:01:33.262435 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.262411 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d106684-1c5e-4bab-822f-1d492dea0f31-kube-api-access-sdtt5" (OuterVolumeSpecName: "kube-api-access-sdtt5") pod "6d106684-1c5e-4bab-822f-1d492dea0f31" (UID: "6d106684-1c5e-4bab-822f-1d492dea0f31"). InnerVolumeSpecName "kube-api-access-sdtt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:01:33.361846 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.361816 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d106684-1c5e-4bab-822f-1d492dea0f31-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:01:33.361846 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.361840 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdtt5\" (UniqueName: \"kubernetes.io/projected/6d106684-1c5e-4bab-822f-1d492dea0f31-kube-api-access-sdtt5\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:01:33.886602 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.886568 2576 generic.go:356] "Generic (PLEG): container finished" podID="7c78217e-49fa-4d15-bec6-365ef8d35f4e" containerID="c2de1559c1a0651fcc789a264d6f18933f63a1593f68f916db11ee86b7748105" exitCode=0 Sep 05 08:01:33.886933 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.886674 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" Sep 05 08:01:33.888362 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.888336 2576 generic.go:356] "Generic (PLEG): container finished" podID="6d106684-1c5e-4bab-822f-1d492dea0f31" containerID="fd0d0ce7ec450548896e7d23959caf855c7f504629f6cd18afb2f90bef045dce" exitCode=0 Sep 05 08:01:33.888483 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.888454 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" Sep 05 08:01:33.893378 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.893353 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" event={"ID":"7c78217e-49fa-4d15-bec6-365ef8d35f4e","Type":"ContainerDied","Data":"c2de1559c1a0651fcc789a264d6f18933f63a1593f68f916db11ee86b7748105"} Sep 05 08:01:33.893472 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.893403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74" event={"ID":"7c78217e-49fa-4d15-bec6-365ef8d35f4e","Type":"ContainerDied","Data":"9fb5bf539c735dae3e08f450ca96627062b4da9f17ffb7ac501a614a89fb1c14"} Sep 05 08:01:33.893472 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.893426 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" event={"ID":"6d106684-1c5e-4bab-822f-1d492dea0f31","Type":"ContainerDied","Data":"fd0d0ce7ec450548896e7d23959caf855c7f504629f6cd18afb2f90bef045dce"} Sep 05 08:01:33.893472 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.893448 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x" event={"ID":"6d106684-1c5e-4bab-822f-1d492dea0f31","Type":"ContainerDied","Data":"e81f30ddb2d9b86a6b471c77ee4e3e32dc8a3bc318ce50d81271db6c43a0019a"} Sep 05 08:01:33.893568 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.893474 2576 scope.go:117] "RemoveContainer" containerID="c2de1559c1a0651fcc789a264d6f18933f63a1593f68f916db11ee86b7748105" Sep 05 08:01:33.904174 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.904159 2576 scope.go:117] "RemoveContainer" containerID="c2de1559c1a0651fcc789a264d6f18933f63a1593f68f916db11ee86b7748105" Sep 05 08:01:33.904473 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:01:33.904455 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2de1559c1a0651fcc789a264d6f18933f63a1593f68f916db11ee86b7748105\": container with ID starting with c2de1559c1a0651fcc789a264d6f18933f63a1593f68f916db11ee86b7748105 not found: ID does not exist" containerID="c2de1559c1a0651fcc789a264d6f18933f63a1593f68f916db11ee86b7748105" Sep 05 08:01:33.904536 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.904480 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2de1559c1a0651fcc789a264d6f18933f63a1593f68f916db11ee86b7748105"} err="failed to get container status \"c2de1559c1a0651fcc789a264d6f18933f63a1593f68f916db11ee86b7748105\": rpc error: code = NotFound desc = could not find container \"c2de1559c1a0651fcc789a264d6f18933f63a1593f68f916db11ee86b7748105\": container with ID starting with c2de1559c1a0651fcc789a264d6f18933f63a1593f68f916db11ee86b7748105 not found: ID does not exist" Sep 05 08:01:33.904536 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.904498 2576 scope.go:117] "RemoveContainer" containerID="fd0d0ce7ec450548896e7d23959caf855c7f504629f6cd18afb2f90bef045dce" Sep 05 08:01:33.913400 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.913383 2576 scope.go:117] "RemoveContainer" containerID="fd0d0ce7ec450548896e7d23959caf855c7f504629f6cd18afb2f90bef045dce" Sep 05 08:01:33.913726 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:01:33.913703 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0d0ce7ec450548896e7d23959caf855c7f504629f6cd18afb2f90bef045dce\": container with ID starting with fd0d0ce7ec450548896e7d23959caf855c7f504629f6cd18afb2f90bef045dce not found: ID does not exist" containerID="fd0d0ce7ec450548896e7d23959caf855c7f504629f6cd18afb2f90bef045dce" Sep 05 08:01:33.913779 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.913737 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0d0ce7ec450548896e7d23959caf855c7f504629f6cd18afb2f90bef045dce"} err="failed to get container status \"fd0d0ce7ec450548896e7d23959caf855c7f504629f6cd18afb2f90bef045dce\": rpc error: code = NotFound desc = could not find container \"fd0d0ce7ec450548896e7d23959caf855c7f504629f6cd18afb2f90bef045dce\": container with ID starting with fd0d0ce7ec450548896e7d23959caf855c7f504629f6cd18afb2f90bef045dce not found: ID does not exist" Sep 05 08:01:33.915109 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.915088 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74"] Sep 05 08:01:33.921029 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.920998 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-73e112cd-7df759d875-8hw74"] Sep 05 08:01:33.934473 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.934453 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x"] Sep 05 08:01:33.937766 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:33.937744 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-73e112cd-794d78bc8b-dw47x"] Sep 05 08:01:35.886804 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:35.886765 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d106684-1c5e-4bab-822f-1d492dea0f31" path="/var/lib/kubelet/pods/6d106684-1c5e-4bab-822f-1d492dea0f31/volumes" Sep 05 08:01:35.887321 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:35.887287 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c78217e-49fa-4d15-bec6-365ef8d35f4e" path="/var/lib/kubelet/pods/7c78217e-49fa-4d15-bec6-365ef8d35f4e/volumes" Sep 05 08:01:38.151318 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.151293 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5"] Sep 05 08:01:38.152165 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.152119 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d106684-1c5e-4bab-822f-1d492dea0f31" containerName="payload-pre-processing" Sep 05 08:01:38.152165 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.152166 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d106684-1c5e-4bab-822f-1d492dea0f31" containerName="payload-pre-processing" Sep 05 08:01:38.152376 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.152191 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c78217e-49fa-4d15-bec6-365ef8d35f4e" containerName="payload-processing" Sep 05 08:01:38.152376 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.152235 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c78217e-49fa-4d15-bec6-365ef8d35f4e" containerName="payload-processing" Sep 05 08:01:38.152551 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.152533 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c78217e-49fa-4d15-bec6-365ef8d35f4e" containerName="payload-processing" Sep 05 08:01:38.152611 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.152567 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d106684-1c5e-4bab-822f-1d492dea0f31" containerName="payload-pre-processing" Sep 05 08:01:38.158246 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.158154 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" Sep 05 08:01:38.163144 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.163117 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7dd976de-dockercfg-2svk2\"" Sep 05 08:01:38.163280 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.163154 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7dd976de\"" Sep 05 08:01:38.166157 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.166132 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5"] Sep 05 08:01:38.180790 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.180767 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws"] Sep 05 08:01:38.183994 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.183968 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" Sep 05 08:01:38.200833 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.200808 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws"] Sep 05 08:01:38.204091 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.204063 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cllg9\" (UniqueName: \"kubernetes.io/projected/ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a-kube-api-access-cllg9\") pod \"payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5\" (UID: \"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" Sep 05 08:01:38.204236 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.204107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5\" (UID: \"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" Sep 05 08:01:38.204236 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.204142 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9199abe8-2c58-43d5-9633-2282e8cdcec0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws\" (UID: \"9199abe8-2c58-43d5-9633-2282e8cdcec0\") " pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" Sep 05 08:01:38.204236 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.204168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8q2z\" (UniqueName: \"kubernetes.io/projected/9199abe8-2c58-43d5-9633-2282e8cdcec0-kube-api-access-c8q2z\") pod \"payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws\" (UID: \"9199abe8-2c58-43d5-9633-2282e8cdcec0\") " pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" Sep 05 08:01:38.304931 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.304896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cllg9\" (UniqueName: \"kubernetes.io/projected/ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a-kube-api-access-cllg9\") pod \"payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5\" (UID: \"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" Sep 05 08:01:38.304931 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.304931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5\" (UID: \"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" Sep 05 08:01:38.305127 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.304951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9199abe8-2c58-43d5-9633-2282e8cdcec0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws\" (UID: \"9199abe8-2c58-43d5-9633-2282e8cdcec0\") " pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" Sep 05 08:01:38.305127 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.304979 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8q2z\" (UniqueName: \"kubernetes.io/projected/9199abe8-2c58-43d5-9633-2282e8cdcec0-kube-api-access-c8q2z\") pod \"payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws\" (UID: \"9199abe8-2c58-43d5-9633-2282e8cdcec0\") " pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" Sep 05 08:01:38.305681 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.305660 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9199abe8-2c58-43d5-9633-2282e8cdcec0-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws\" (UID: \"9199abe8-2c58-43d5-9633-2282e8cdcec0\") " pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" Sep 05 08:01:38.305740 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.305669 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5\" (UID: \"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" Sep 05 08:01:38.317063 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.317045 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8q2z\" (UniqueName: \"kubernetes.io/projected/9199abe8-2c58-43d5-9633-2282e8cdcec0-kube-api-access-c8q2z\") pod \"payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws\" (UID: \"9199abe8-2c58-43d5-9633-2282e8cdcec0\") " pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" Sep 05 08:01:38.317290 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.317269 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cllg9\" (UniqueName: \"kubernetes.io/projected/ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a-kube-api-access-cllg9\") pod \"payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5\" (UID: \"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" Sep 05 08:01:38.471337 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.471262 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" Sep 05 08:01:38.494557 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.494530 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" Sep 05 08:01:38.756668 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.756640 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5"] Sep 05 08:01:38.758227 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:01:38.758187 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce6146a7_edd4_44ce_8338_1ec7f1ca0a4a.slice/crio-daaf7fed40eeef47948a85d73e1d8ae4f4d5895a6ee7e21c8e9f2a35a83c8da9 WatchSource:0}: Error finding container daaf7fed40eeef47948a85d73e1d8ae4f4d5895a6ee7e21c8e9f2a35a83c8da9: Status 404 returned error can't find the container with id daaf7fed40eeef47948a85d73e1d8ae4f4d5895a6ee7e21c8e9f2a35a83c8da9 Sep 05 08:01:38.784817 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.784796 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws"] Sep 05 08:01:38.790180 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:01:38.790157 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9199abe8_2c58_43d5_9633_2282e8cdcec0.slice/crio-381498779a2faf1e010a4be4b94acb6593f1b5a81dd92c8678f478cc3f8bff8f WatchSource:0}: Error finding container 381498779a2faf1e010a4be4b94acb6593f1b5a81dd92c8678f478cc3f8bff8f: Status 404 returned error can't find the container with id 381498779a2faf1e010a4be4b94acb6593f1b5a81dd92c8678f478cc3f8bff8f Sep 05 08:01:38.921315 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.921278 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" event={"ID":"9199abe8-2c58-43d5-9633-2282e8cdcec0","Type":"ContainerStarted","Data":"0b3eca93df4da0f41fa0bbc3431f29ef1386c3c6f913547ef89579452decb409"} Sep 05 08:01:38.921494 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.921335 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" event={"ID":"9199abe8-2c58-43d5-9633-2282e8cdcec0","Type":"ContainerStarted","Data":"381498779a2faf1e010a4be4b94acb6593f1b5a81dd92c8678f478cc3f8bff8f"} Sep 05 08:01:38.921494 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.921387 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" Sep 05 08:01:38.923131 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.923093 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" event={"ID":"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a","Type":"ContainerStarted","Data":"c0847b3c096bcddd9c9dca24ea153943d635d1976c766feb4b7f4d5ffd358839"} Sep 05 08:01:38.923274 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.923160 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" Sep 05 08:01:38.923274 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.923177 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" event={"ID":"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a","Type":"ContainerStarted","Data":"daaf7fed40eeef47948a85d73e1d8ae4f4d5895a6ee7e21c8e9f2a35a83c8da9"} Sep 05 08:01:38.943382 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.943339 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" podStartSLOduration=0.943325282 podStartE2EDuration="943.325282ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:01:38.940774707 +0000 UTC m=+801.549749829" watchObservedRunningTime="2026-09-05 08:01:38.943325282 +0000 UTC m=+801.552300465" Sep 05 08:01:38.960154 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:38.960113 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" podStartSLOduration=0.960097132 podStartE2EDuration="960.097132ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:01:38.958841028 +0000 UTC m=+801.567816154" watchObservedRunningTime="2026-09-05 08:01:38.960097132 +0000 UTC m=+801.569072256" Sep 05 08:01:44.252024 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:44.251988 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:01:49.933808 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:49.933777 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" Sep 05 08:01:49.934232 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:01:49.933855 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" Sep 05 08:02:00.381051 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:00.381015 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:02:08.877925 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:08.877882 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws"] Sep 05 08:02:08.878537 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:08.878153 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" podUID="9199abe8-2c58-43d5-9633-2282e8cdcec0" containerName="payload-processing" containerID="cri-o://0b3eca93df4da0f41fa0bbc3431f29ef1386c3c6f913547ef89579452decb409" gracePeriod=30 Sep 05 08:02:08.890267 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:08.890088 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5"] Sep 05 08:02:08.890797 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:08.890578 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" podUID="ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a" containerName="payload-pre-processing" containerID="cri-o://c0847b3c096bcddd9c9dca24ea153943d635d1976c766feb4b7f4d5ffd358839" gracePeriod=30 Sep 05 08:02:09.100034 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.099995 2576 generic.go:356] "Generic (PLEG): container finished" podID="9199abe8-2c58-43d5-9633-2282e8cdcec0" containerID="0b3eca93df4da0f41fa0bbc3431f29ef1386c3c6f913547ef89579452decb409" exitCode=0 Sep 05 08:02:09.100178 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.100094 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" event={"ID":"9199abe8-2c58-43d5-9633-2282e8cdcec0","Type":"ContainerDied","Data":"0b3eca93df4da0f41fa0bbc3431f29ef1386c3c6f913547ef89579452decb409"} Sep 05 08:02:09.102166 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.102140 2576 generic.go:356] "Generic (PLEG): container finished" podID="ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a" containerID="c0847b3c096bcddd9c9dca24ea153943d635d1976c766feb4b7f4d5ffd358839" exitCode=0 Sep 05 08:02:09.102365 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.102347 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" event={"ID":"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a","Type":"ContainerDied","Data":"c0847b3c096bcddd9c9dca24ea153943d635d1976c766feb4b7f4d5ffd358839"} Sep 05 08:02:09.164583 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.164557 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" Sep 05 08:02:09.167277 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.167257 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" Sep 05 08:02:09.280984 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.280934 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cllg9\" (UniqueName: \"kubernetes.io/projected/ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a-kube-api-access-cllg9\") pod \"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a\" (UID: \"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a\") " Sep 05 08:02:09.281163 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.281027 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8q2z\" (UniqueName: \"kubernetes.io/projected/9199abe8-2c58-43d5-9633-2282e8cdcec0-kube-api-access-c8q2z\") pod \"9199abe8-2c58-43d5-9633-2282e8cdcec0\" (UID: \"9199abe8-2c58-43d5-9633-2282e8cdcec0\") " Sep 05 08:02:09.281163 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.281067 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a-plugins-config-volume\") pod \"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a\" (UID: \"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a\") " Sep 05 08:02:09.281163 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.281114 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9199abe8-2c58-43d5-9633-2282e8cdcec0-plugins-config-volume\") pod \"9199abe8-2c58-43d5-9633-2282e8cdcec0\" (UID: \"9199abe8-2c58-43d5-9633-2282e8cdcec0\") " Sep 05 08:02:09.281492 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.281470 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a" (UID: "ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:02:09.281569 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.281529 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9199abe8-2c58-43d5-9633-2282e8cdcec0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9199abe8-2c58-43d5-9633-2282e8cdcec0" (UID: "9199abe8-2c58-43d5-9633-2282e8cdcec0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:02:09.283254 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.283232 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a-kube-api-access-cllg9" (OuterVolumeSpecName: "kube-api-access-cllg9") pod "ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a" (UID: "ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a"). InnerVolumeSpecName "kube-api-access-cllg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:02:09.283254 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.283237 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9199abe8-2c58-43d5-9633-2282e8cdcec0-kube-api-access-c8q2z" (OuterVolumeSpecName: "kube-api-access-c8q2z") pod "9199abe8-2c58-43d5-9633-2282e8cdcec0" (UID: "9199abe8-2c58-43d5-9633-2282e8cdcec0"). InnerVolumeSpecName "kube-api-access-c8q2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:02:09.382205 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.382168 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:02:09.382205 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.382197 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9199abe8-2c58-43d5-9633-2282e8cdcec0-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:02:09.382428 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.382230 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cllg9\" (UniqueName: \"kubernetes.io/projected/ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a-kube-api-access-cllg9\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:02:09.382428 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:09.382241 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8q2z\" (UniqueName: \"kubernetes.io/projected/9199abe8-2c58-43d5-9633-2282e8cdcec0-kube-api-access-c8q2z\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:02:10.110097 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:10.110059 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" event={"ID":"9199abe8-2c58-43d5-9633-2282e8cdcec0","Type":"ContainerDied","Data":"381498779a2faf1e010a4be4b94acb6593f1b5a81dd92c8678f478cc3f8bff8f"} Sep 05 08:02:10.110617 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:10.110137 2576 scope.go:117] "RemoveContainer" containerID="0b3eca93df4da0f41fa0bbc3431f29ef1386c3c6f913547ef89579452decb409" Sep 05 08:02:10.110617 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:10.110069 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws" Sep 05 08:02:10.111970 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:10.111943 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" event={"ID":"ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a","Type":"ContainerDied","Data":"daaf7fed40eeef47948a85d73e1d8ae4f4d5895a6ee7e21c8e9f2a35a83c8da9"} Sep 05 08:02:10.112036 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:10.112014 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5" Sep 05 08:02:10.122942 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:10.122924 2576 scope.go:117] "RemoveContainer" containerID="c0847b3c096bcddd9c9dca24ea153943d635d1976c766feb4b7f4d5ffd358839" Sep 05 08:02:10.158366 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:10.158288 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws"] Sep 05 08:02:10.169257 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:10.169226 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7dd976de-566c5bffc8-8f9ws"] Sep 05 08:02:10.193875 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:10.193846 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5"] Sep 05 08:02:10.200464 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:10.200432 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7dd976de-7649848fdb-gp9q5"] Sep 05 08:02:11.886200 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:11.886165 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9199abe8-2c58-43d5-9633-2282e8cdcec0" path="/var/lib/kubelet/pods/9199abe8-2c58-43d5-9633-2282e8cdcec0/volumes" Sep 05 08:02:11.886714 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:11.886694 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a" path="/var/lib/kubelet/pods/ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a/volumes" Sep 05 08:02:14.147603 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.147572 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl"] Sep 05 08:02:14.147985 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.147926 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9199abe8-2c58-43d5-9633-2282e8cdcec0" containerName="payload-processing" Sep 05 08:02:14.147985 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.147938 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9199abe8-2c58-43d5-9633-2282e8cdcec0" containerName="payload-processing" Sep 05 08:02:14.147985 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.147951 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a" containerName="payload-pre-processing" Sep 05 08:02:14.147985 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.147956 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a" containerName="payload-pre-processing" Sep 05 08:02:14.148118 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.148020 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9199abe8-2c58-43d5-9633-2282e8cdcec0" containerName="payload-processing" Sep 05 08:02:14.148118 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.148030 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce6146a7-edd4-44ce-8338-1ec7f1ca0a4a" containerName="payload-pre-processing" Sep 05 08:02:14.151120 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.151102 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" Sep 05 08:02:14.154248 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.154227 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-29f90b97\"" Sep 05 08:02:14.154396 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.154330 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-29f90b97-dockercfg-8l2vr\"" Sep 05 08:02:14.162483 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.162461 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl"] Sep 05 08:02:14.211273 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.211242 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9"] Sep 05 08:02:14.214668 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.214652 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" Sep 05 08:02:14.228140 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.228118 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9"] Sep 05 08:02:14.323456 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.323425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n2gt\" (UniqueName: \"kubernetes.io/projected/552591a4-dc27-4ee1-8b2c-2282d397c1a4-kube-api-access-9n2gt\") pod \"payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl\" (UID: \"552591a4-dc27-4ee1-8b2c-2282d397c1a4\") " pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" Sep 05 08:02:14.323661 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.323470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbbx7\" (UniqueName: \"kubernetes.io/projected/a2829f20-faf4-4954-b296-713d51578985-kube-api-access-vbbx7\") pod \"payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9\" (UID: \"a2829f20-faf4-4954-b296-713d51578985\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" Sep 05 08:02:14.323661 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.323580 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2829f20-faf4-4954-b296-713d51578985-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9\" (UID: \"a2829f20-faf4-4954-b296-713d51578985\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" Sep 05 08:02:14.323661 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.323608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552591a4-dc27-4ee1-8b2c-2282d397c1a4-plugins-config-volume\") pod \"payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl\" (UID: \"552591a4-dc27-4ee1-8b2c-2282d397c1a4\") " pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" Sep 05 08:02:14.424630 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.424538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2gt\" (UniqueName: \"kubernetes.io/projected/552591a4-dc27-4ee1-8b2c-2282d397c1a4-kube-api-access-9n2gt\") pod \"payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl\" (UID: \"552591a4-dc27-4ee1-8b2c-2282d397c1a4\") " pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" Sep 05 08:02:14.424630 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.424598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbbx7\" (UniqueName: \"kubernetes.io/projected/a2829f20-faf4-4954-b296-713d51578985-kube-api-access-vbbx7\") pod \"payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9\" (UID: \"a2829f20-faf4-4954-b296-713d51578985\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" Sep 05 08:02:14.424910 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.424653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2829f20-faf4-4954-b296-713d51578985-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9\" (UID: \"a2829f20-faf4-4954-b296-713d51578985\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" Sep 05 08:02:14.424910 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.424670 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552591a4-dc27-4ee1-8b2c-2282d397c1a4-plugins-config-volume\") pod \"payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl\" (UID: \"552591a4-dc27-4ee1-8b2c-2282d397c1a4\") " pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" Sep 05 08:02:14.425352 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.425335 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552591a4-dc27-4ee1-8b2c-2282d397c1a4-plugins-config-volume\") pod \"payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl\" (UID: \"552591a4-dc27-4ee1-8b2c-2282d397c1a4\") " pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" Sep 05 08:02:14.425407 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.425355 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2829f20-faf4-4954-b296-713d51578985-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9\" (UID: \"a2829f20-faf4-4954-b296-713d51578985\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" Sep 05 08:02:14.433949 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.433927 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbbx7\" (UniqueName: \"kubernetes.io/projected/a2829f20-faf4-4954-b296-713d51578985-kube-api-access-vbbx7\") pod \"payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9\" (UID: \"a2829f20-faf4-4954-b296-713d51578985\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" Sep 05 08:02:14.435541 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.435519 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n2gt\" (UniqueName: \"kubernetes.io/projected/552591a4-dc27-4ee1-8b2c-2282d397c1a4-kube-api-access-9n2gt\") pod \"payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl\" (UID: \"552591a4-dc27-4ee1-8b2c-2282d397c1a4\") " pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" Sep 05 08:02:14.461474 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.461448 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" Sep 05 08:02:14.524385 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.524350 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" Sep 05 08:02:14.737522 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.737498 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl"] Sep 05 08:02:14.739044 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:02:14.739019 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552591a4_dc27_4ee1_8b2c_2282d397c1a4.slice/crio-1db4f1aedcac173c917596b21a141623857c9484639bf26120619ff61f4a57fe WatchSource:0}: Error finding container 1db4f1aedcac173c917596b21a141623857c9484639bf26120619ff61f4a57fe: Status 404 returned error can't find the container with id 1db4f1aedcac173c917596b21a141623857c9484639bf26120619ff61f4a57fe Sep 05 08:02:14.802360 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:14.802335 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9"] Sep 05 08:02:14.803401 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:02:14.803378 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2829f20_faf4_4954_b296_713d51578985.slice/crio-e70e773e7ab253622dc999791bfc1d5178baaa420fe8b35ec122fdf03b757b2e WatchSource:0}: Error finding container e70e773e7ab253622dc999791bfc1d5178baaa420fe8b35ec122fdf03b757b2e: Status 404 returned error can't find the container with id e70e773e7ab253622dc999791bfc1d5178baaa420fe8b35ec122fdf03b757b2e Sep 05 08:02:15.059221 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:15.059129 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:02:15.151070 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:15.151032 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" event={"ID":"a2829f20-faf4-4954-b296-713d51578985","Type":"ContainerStarted","Data":"b996579121214944733766a3000bf0aa42f9a7931a66181a91eba48822e424c1"} Sep 05 08:02:15.151572 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:15.151096 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" event={"ID":"a2829f20-faf4-4954-b296-713d51578985","Type":"ContainerStarted","Data":"e70e773e7ab253622dc999791bfc1d5178baaa420fe8b35ec122fdf03b757b2e"} Sep 05 08:02:15.151572 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:15.151222 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" Sep 05 08:02:15.152765 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:15.152743 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" event={"ID":"552591a4-dc27-4ee1-8b2c-2282d397c1a4","Type":"ContainerStarted","Data":"960fb0fc380d00ed5783e915801cd1c457c47a6d6857ad0bcc621e5a1f01641e"} Sep 05 08:02:15.152889 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:15.152780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" event={"ID":"552591a4-dc27-4ee1-8b2c-2282d397c1a4","Type":"ContainerStarted","Data":"1db4f1aedcac173c917596b21a141623857c9484639bf26120619ff61f4a57fe"} Sep 05 08:02:15.152889 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:15.152807 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" Sep 05 08:02:15.176778 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:15.176726 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" podStartSLOduration=1.176708603 podStartE2EDuration="1.176708603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:02:15.176341178 +0000 UTC m=+837.785316303" watchObservedRunningTime="2026-09-05 08:02:15.176708603 +0000 UTC m=+837.785683728" Sep 05 08:02:15.195135 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:15.195087 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" podStartSLOduration=1.1950725389999999 podStartE2EDuration="1.195072539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:02:15.193777739 +0000 UTC m=+837.802752863" watchObservedRunningTime="2026-09-05 08:02:15.195072539 +0000 UTC m=+837.804047662" Sep 05 08:02:26.160881 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:26.160848 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" Sep 05 08:02:26.161406 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:26.161388 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" Sep 05 08:02:32.562200 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:32.562161 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:02:44.073694 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.073656 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl"] Sep 05 08:02:44.074289 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.073899 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" podUID="552591a4-dc27-4ee1-8b2c-2282d397c1a4" containerName="payload-processing" containerID="cri-o://960fb0fc380d00ed5783e915801cd1c457c47a6d6857ad0bcc621e5a1f01641e" gracePeriod=30 Sep 05 08:02:44.085258 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.085180 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9"] Sep 05 08:02:44.085534 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.085508 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" podUID="a2829f20-faf4-4954-b296-713d51578985" containerName="payload-pre-processing" containerID="cri-o://b996579121214944733766a3000bf0aa42f9a7931a66181a91eba48822e424c1" gracePeriod=30 Sep 05 08:02:44.317054 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.317020 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2829f20-faf4-4954-b296-713d51578985" containerID="b996579121214944733766a3000bf0aa42f9a7931a66181a91eba48822e424c1" exitCode=0 Sep 05 08:02:44.317240 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.317103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" event={"ID":"a2829f20-faf4-4954-b296-713d51578985","Type":"ContainerDied","Data":"b996579121214944733766a3000bf0aa42f9a7931a66181a91eba48822e424c1"} Sep 05 08:02:44.319267 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.319240 2576 generic.go:356] "Generic (PLEG): container finished" podID="552591a4-dc27-4ee1-8b2c-2282d397c1a4" containerID="960fb0fc380d00ed5783e915801cd1c457c47a6d6857ad0bcc621e5a1f01641e" exitCode=0 Sep 05 08:02:44.319402 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.319261 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" event={"ID":"552591a4-dc27-4ee1-8b2c-2282d397c1a4","Type":"ContainerDied","Data":"960fb0fc380d00ed5783e915801cd1c457c47a6d6857ad0bcc621e5a1f01641e"} Sep 05 08:02:44.347000 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.346979 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" Sep 05 08:02:44.373066 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.373046 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" Sep 05 08:02:44.394271 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.394248 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552591a4-dc27-4ee1-8b2c-2282d397c1a4-plugins-config-volume\") pod \"552591a4-dc27-4ee1-8b2c-2282d397c1a4\" (UID: \"552591a4-dc27-4ee1-8b2c-2282d397c1a4\") " Sep 05 08:02:44.394858 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.394835 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n2gt\" (UniqueName: \"kubernetes.io/projected/552591a4-dc27-4ee1-8b2c-2282d397c1a4-kube-api-access-9n2gt\") pod \"552591a4-dc27-4ee1-8b2c-2282d397c1a4\" (UID: \"552591a4-dc27-4ee1-8b2c-2282d397c1a4\") " Sep 05 08:02:44.394978 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.394731 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552591a4-dc27-4ee1-8b2c-2282d397c1a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "552591a4-dc27-4ee1-8b2c-2282d397c1a4" (UID: "552591a4-dc27-4ee1-8b2c-2282d397c1a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:02:44.395132 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.395106 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552591a4-dc27-4ee1-8b2c-2282d397c1a4-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:02:44.397290 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.397266 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552591a4-dc27-4ee1-8b2c-2282d397c1a4-kube-api-access-9n2gt" (OuterVolumeSpecName: "kube-api-access-9n2gt") pod "552591a4-dc27-4ee1-8b2c-2282d397c1a4" (UID: "552591a4-dc27-4ee1-8b2c-2282d397c1a4"). InnerVolumeSpecName "kube-api-access-9n2gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:02:44.495596 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.495565 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbbx7\" (UniqueName: \"kubernetes.io/projected/a2829f20-faf4-4954-b296-713d51578985-kube-api-access-vbbx7\") pod \"a2829f20-faf4-4954-b296-713d51578985\" (UID: \"a2829f20-faf4-4954-b296-713d51578985\") " Sep 05 08:02:44.495765 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.495628 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2829f20-faf4-4954-b296-713d51578985-plugins-config-volume\") pod \"a2829f20-faf4-4954-b296-713d51578985\" (UID: \"a2829f20-faf4-4954-b296-713d51578985\") " Sep 05 08:02:44.495765 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.495761 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n2gt\" (UniqueName: \"kubernetes.io/projected/552591a4-dc27-4ee1-8b2c-2282d397c1a4-kube-api-access-9n2gt\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:02:44.496093 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.496058 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2829f20-faf4-4954-b296-713d51578985-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2829f20-faf4-4954-b296-713d51578985" (UID: "a2829f20-faf4-4954-b296-713d51578985"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:02:44.497764 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.497744 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2829f20-faf4-4954-b296-713d51578985-kube-api-access-vbbx7" (OuterVolumeSpecName: "kube-api-access-vbbx7") pod "a2829f20-faf4-4954-b296-713d51578985" (UID: "a2829f20-faf4-4954-b296-713d51578985"). InnerVolumeSpecName "kube-api-access-vbbx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:02:44.596913 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.596831 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2829f20-faf4-4954-b296-713d51578985-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:02:44.596913 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:44.596860 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbbx7\" (UniqueName: \"kubernetes.io/projected/a2829f20-faf4-4954-b296-713d51578985-kube-api-access-vbbx7\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:02:45.325638 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:45.325604 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" Sep 05 08:02:45.325638 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:45.325625 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9" event={"ID":"a2829f20-faf4-4954-b296-713d51578985","Type":"ContainerDied","Data":"e70e773e7ab253622dc999791bfc1d5178baaa420fe8b35ec122fdf03b757b2e"} Sep 05 08:02:45.326128 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:45.325689 2576 scope.go:117] "RemoveContainer" containerID="b996579121214944733766a3000bf0aa42f9a7931a66181a91eba48822e424c1" Sep 05 08:02:45.327080 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:45.327022 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" Sep 05 08:02:45.327080 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:45.327042 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl" event={"ID":"552591a4-dc27-4ee1-8b2c-2282d397c1a4","Type":"ContainerDied","Data":"1db4f1aedcac173c917596b21a141623857c9484639bf26120619ff61f4a57fe"} Sep 05 08:02:45.335443 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:45.335427 2576 scope.go:117] "RemoveContainer" containerID="960fb0fc380d00ed5783e915801cd1c457c47a6d6857ad0bcc621e5a1f01641e" Sep 05 08:02:45.351126 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:45.351103 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl"] Sep 05 08:02:45.353487 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:45.353468 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-29f90b97-6cc6c5ff-5sghl"] Sep 05 08:02:45.362927 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:45.362908 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9"] Sep 05 08:02:45.367534 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:45.367515 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-29f90b97-588cc7d586-mhbk9"] Sep 05 08:02:45.886248 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:45.886190 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="552591a4-dc27-4ee1-8b2c-2282d397c1a4" path="/var/lib/kubelet/pods/552591a4-dc27-4ee1-8b2c-2282d397c1a4/volumes" Sep 05 08:02:45.886646 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:45.886625 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2829f20-faf4-4954-b296-713d51578985" path="/var/lib/kubelet/pods/a2829f20-faf4-4954-b296-713d51578985/volumes" Sep 05 08:02:50.279502 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.279459 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:02:50.699279 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.699249 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn"] Sep 05 08:02:50.699631 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.699618 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2829f20-faf4-4954-b296-713d51578985" containerName="payload-pre-processing" Sep 05 08:02:50.699683 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.699632 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2829f20-faf4-4954-b296-713d51578985" containerName="payload-pre-processing" Sep 05 08:02:50.699683 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.699653 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="552591a4-dc27-4ee1-8b2c-2282d397c1a4" containerName="payload-processing" Sep 05 08:02:50.699683 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.699659 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="552591a4-dc27-4ee1-8b2c-2282d397c1a4" containerName="payload-processing" Sep 05 08:02:50.699776 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.699715 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2829f20-faf4-4954-b296-713d51578985" containerName="payload-pre-processing" Sep 05 08:02:50.699776 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.699724 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="552591a4-dc27-4ee1-8b2c-2282d397c1a4" containerName="payload-processing" Sep 05 08:02:50.704005 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.703984 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" Sep 05 08:02:50.707139 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.707115 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-46865726\"" Sep 05 08:02:50.718302 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.718276 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn"] Sep 05 08:02:50.750090 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.750065 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a89ca1-eb15-4e5d-9c48-9772bd709620-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn\" (UID: \"d2a89ca1-eb15-4e5d-9c48-9772bd709620\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" Sep 05 08:02:50.750255 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.750223 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5gn9\" (UniqueName: \"kubernetes.io/projected/d2a89ca1-eb15-4e5d-9c48-9772bd709620-kube-api-access-l5gn9\") pod \"payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn\" (UID: \"d2a89ca1-eb15-4e5d-9c48-9772bd709620\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" Sep 05 08:02:50.847844 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.847816 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh"] Sep 05 08:02:50.851359 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.851335 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" Sep 05 08:02:50.851484 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.851354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a89ca1-eb15-4e5d-9c48-9772bd709620-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn\" (UID: \"d2a89ca1-eb15-4e5d-9c48-9772bd709620\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" Sep 05 08:02:50.851484 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.851453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5gn9\" (UniqueName: \"kubernetes.io/projected/d2a89ca1-eb15-4e5d-9c48-9772bd709620-kube-api-access-l5gn9\") pod \"payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn\" (UID: \"d2a89ca1-eb15-4e5d-9c48-9772bd709620\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" Sep 05 08:02:50.852045 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.852026 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a89ca1-eb15-4e5d-9c48-9772bd709620-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn\" (UID: \"d2a89ca1-eb15-4e5d-9c48-9772bd709620\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" Sep 05 08:02:50.854329 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.854311 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-46865726-dockercfg-ngh4l\"" Sep 05 08:02:50.863458 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.863436 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh"] Sep 05 08:02:50.864790 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.864768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5gn9\" (UniqueName: \"kubernetes.io/projected/d2a89ca1-eb15-4e5d-9c48-9772bd709620-kube-api-access-l5gn9\") pod \"payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn\" (UID: \"d2a89ca1-eb15-4e5d-9c48-9772bd709620\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" Sep 05 08:02:50.954595 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.954497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6efcb5-8ec1-4635-91b9-b6573a869a47-plugins-config-volume\") pod \"payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh\" (UID: \"9d6efcb5-8ec1-4635-91b9-b6573a869a47\") " pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" Sep 05 08:02:50.954755 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:50.954604 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gzp\" (UniqueName: \"kubernetes.io/projected/9d6efcb5-8ec1-4635-91b9-b6573a869a47-kube-api-access-j4gzp\") pod \"payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh\" (UID: \"9d6efcb5-8ec1-4635-91b9-b6573a869a47\") " pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" Sep 05 08:02:51.014512 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:51.014481 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" Sep 05 08:02:51.055710 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:51.055676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6efcb5-8ec1-4635-91b9-b6573a869a47-plugins-config-volume\") pod \"payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh\" (UID: \"9d6efcb5-8ec1-4635-91b9-b6573a869a47\") " pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" Sep 05 08:02:51.055901 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:51.055759 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gzp\" (UniqueName: \"kubernetes.io/projected/9d6efcb5-8ec1-4635-91b9-b6573a869a47-kube-api-access-j4gzp\") pod \"payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh\" (UID: \"9d6efcb5-8ec1-4635-91b9-b6573a869a47\") " pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" Sep 05 08:02:51.056627 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:51.056599 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6efcb5-8ec1-4635-91b9-b6573a869a47-plugins-config-volume\") pod \"payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh\" (UID: \"9d6efcb5-8ec1-4635-91b9-b6573a869a47\") " pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" Sep 05 08:02:51.065840 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:51.065661 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gzp\" (UniqueName: \"kubernetes.io/projected/9d6efcb5-8ec1-4635-91b9-b6573a869a47-kube-api-access-j4gzp\") pod \"payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh\" (UID: \"9d6efcb5-8ec1-4635-91b9-b6573a869a47\") " pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" Sep 05 08:02:51.162486 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:51.162451 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" Sep 05 08:02:51.292045 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:51.292018 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn"] Sep 05 08:02:51.294636 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:02:51.294605 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a89ca1_eb15_4e5d_9c48_9772bd709620.slice/crio-d8ffe504eb4fb4e177908425b13d95d6390c931e6db400c0c0de2fff0f959e9c WatchSource:0}: Error finding container d8ffe504eb4fb4e177908425b13d95d6390c931e6db400c0c0de2fff0f959e9c: Status 404 returned error can't find the container with id d8ffe504eb4fb4e177908425b13d95d6390c931e6db400c0c0de2fff0f959e9c Sep 05 08:02:51.370826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:51.370799 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" event={"ID":"d2a89ca1-eb15-4e5d-9c48-9772bd709620","Type":"ContainerStarted","Data":"217ea1b99754e20be6856543ccb1d277cb3b378bac27f39a49e497831ceee1c6"} Sep 05 08:02:51.370942 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:51.370858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" event={"ID":"d2a89ca1-eb15-4e5d-9c48-9772bd709620","Type":"ContainerStarted","Data":"d8ffe504eb4fb4e177908425b13d95d6390c931e6db400c0c0de2fff0f959e9c"} Sep 05 08:02:51.370942 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:51.370904 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" Sep 05 08:02:51.399413 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:51.399354 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" podStartSLOduration=1.399333597 podStartE2EDuration="1.399333597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:02:51.395937849 +0000 UTC m=+874.004912973" watchObservedRunningTime="2026-09-05 08:02:51.399333597 +0000 UTC m=+874.008308721" Sep 05 08:02:51.444829 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:51.444804 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh"] Sep 05 08:02:51.446946 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:02:51.446916 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d6efcb5_8ec1_4635_91b9_b6573a869a47.slice/crio-0f192fc5c69592a46cb3e8fa9b2e3544ae92b6de5ce59bc1b34f590346125b8c WatchSource:0}: Error finding container 0f192fc5c69592a46cb3e8fa9b2e3544ae92b6de5ce59bc1b34f590346125b8c: Status 404 returned error can't find the container with id 0f192fc5c69592a46cb3e8fa9b2e3544ae92b6de5ce59bc1b34f590346125b8c Sep 05 08:02:52.379108 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:52.379071 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" event={"ID":"9d6efcb5-8ec1-4635-91b9-b6573a869a47","Type":"ContainerStarted","Data":"e5e1ddaa339490851d35b4e70b61ecac4030c6505fd3cdf977c9acf4e338357e"} Sep 05 08:02:52.379550 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:52.379137 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" event={"ID":"9d6efcb5-8ec1-4635-91b9-b6573a869a47","Type":"ContainerStarted","Data":"0f192fc5c69592a46cb3e8fa9b2e3544ae92b6de5ce59bc1b34f590346125b8c"} Sep 05 08:02:52.379550 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:52.379243 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" Sep 05 08:02:52.398649 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:02:52.398586 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" podStartSLOduration=2.398566985 podStartE2EDuration="2.398566985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:02:52.398039491 +0000 UTC m=+875.007014616" watchObservedRunningTime="2026-09-05 08:02:52.398566985 +0000 UTC m=+875.007542108" Sep 05 08:03:02.382238 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:02.382179 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" Sep 05 08:03:03.387898 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:03.387866 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" Sep 05 08:03:03.716031 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:03.715939 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p"] Sep 05 08:03:03.716195 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:03.716170 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" podUID="d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c" containerName="payload-processing" containerID="cri-o://b417b4be96cc5e3064f59b19d1ac9a4faf32d6bd879210f094fd0f67fcb7862f" gracePeriod=30 Sep 05 08:03:03.723946 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:03.723918 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p"] Sep 05 08:03:03.724360 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:03.724153 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" podUID="22270ab3-14df-4da1-be90-ad5d48d59570" containerName="payload-pre-processing" containerID="cri-o://c4a77482001ef1c84e3333ed27be1b84ef8e01626bc45887d7a4de3fde1db3aa" gracePeriod=30 Sep 05 08:03:03.988813 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:03.988787 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" Sep 05 08:03:04.012520 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.012493 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" Sep 05 08:03:04.070790 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.070759 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lklc\" (UniqueName: \"kubernetes.io/projected/d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c-kube-api-access-4lklc\") pod \"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c\" (UID: \"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c\") " Sep 05 08:03:04.070957 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.070798 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c-plugins-config-volume\") pod \"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c\" (UID: \"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c\") " Sep 05 08:03:04.071205 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.071185 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c" (UID: "d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:03:04.073259 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.073235 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c-kube-api-access-4lklc" (OuterVolumeSpecName: "kube-api-access-4lklc") pod "d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c" (UID: "d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c"). InnerVolumeSpecName "kube-api-access-4lklc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:03:04.171762 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.171728 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22270ab3-14df-4da1-be90-ad5d48d59570-plugins-config-volume\") pod \"22270ab3-14df-4da1-be90-ad5d48d59570\" (UID: \"22270ab3-14df-4da1-be90-ad5d48d59570\") " Sep 05 08:03:04.171938 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.171785 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2p7v\" (UniqueName: \"kubernetes.io/projected/22270ab3-14df-4da1-be90-ad5d48d59570-kube-api-access-c2p7v\") pod \"22270ab3-14df-4da1-be90-ad5d48d59570\" (UID: \"22270ab3-14df-4da1-be90-ad5d48d59570\") " Sep 05 08:03:04.171938 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.171929 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lklc\" (UniqueName: \"kubernetes.io/projected/d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c-kube-api-access-4lklc\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:03:04.172012 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.171939 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:03:04.172136 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.172113 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22270ab3-14df-4da1-be90-ad5d48d59570-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "22270ab3-14df-4da1-be90-ad5d48d59570" (UID: "22270ab3-14df-4da1-be90-ad5d48d59570"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:03:04.174040 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.174021 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22270ab3-14df-4da1-be90-ad5d48d59570-kube-api-access-c2p7v" (OuterVolumeSpecName: "kube-api-access-c2p7v") pod "22270ab3-14df-4da1-be90-ad5d48d59570" (UID: "22270ab3-14df-4da1-be90-ad5d48d59570"). InnerVolumeSpecName "kube-api-access-c2p7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:03:04.273073 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.272996 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22270ab3-14df-4da1-be90-ad5d48d59570-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:03:04.273073 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.273021 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2p7v\" (UniqueName: \"kubernetes.io/projected/22270ab3-14df-4da1-be90-ad5d48d59570-kube-api-access-c2p7v\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:03:04.449406 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.449370 2576 generic.go:356] "Generic (PLEG): container finished" podID="22270ab3-14df-4da1-be90-ad5d48d59570" containerID="c4a77482001ef1c84e3333ed27be1b84ef8e01626bc45887d7a4de3fde1db3aa" exitCode=0 Sep 05 08:03:04.449839 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.449414 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" Sep 05 08:03:04.449839 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.449441 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" event={"ID":"22270ab3-14df-4da1-be90-ad5d48d59570","Type":"ContainerDied","Data":"c4a77482001ef1c84e3333ed27be1b84ef8e01626bc45887d7a4de3fde1db3aa"} Sep 05 08:03:04.449839 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.449497 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p" event={"ID":"22270ab3-14df-4da1-be90-ad5d48d59570","Type":"ContainerDied","Data":"4bba1fa83eda9d763b64c8645a1fab2939dd69d495afabdb7af0d361611836a4"} Sep 05 08:03:04.449839 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.449528 2576 scope.go:117] "RemoveContainer" containerID="c4a77482001ef1c84e3333ed27be1b84ef8e01626bc45887d7a4de3fde1db3aa" Sep 05 08:03:04.450964 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.450944 2576 generic.go:356] "Generic (PLEG): container finished" podID="d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c" containerID="b417b4be96cc5e3064f59b19d1ac9a4faf32d6bd879210f094fd0f67fcb7862f" exitCode=0 Sep 05 08:03:04.451071 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.450977 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" Sep 05 08:03:04.451071 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.451020 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" event={"ID":"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c","Type":"ContainerDied","Data":"b417b4be96cc5e3064f59b19d1ac9a4faf32d6bd879210f094fd0f67fcb7862f"} Sep 05 08:03:04.451185 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.451082 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p" event={"ID":"d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c","Type":"ContainerDied","Data":"9904e7caad563548383796c6d64bcffa4531e9bfd40b3681280e316620621cc7"} Sep 05 08:03:04.460000 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.459981 2576 scope.go:117] "RemoveContainer" containerID="c4a77482001ef1c84e3333ed27be1b84ef8e01626bc45887d7a4de3fde1db3aa" Sep 05 08:03:04.460368 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:03:04.460348 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a77482001ef1c84e3333ed27be1b84ef8e01626bc45887d7a4de3fde1db3aa\": container with ID starting with c4a77482001ef1c84e3333ed27be1b84ef8e01626bc45887d7a4de3fde1db3aa not found: ID does not exist" containerID="c4a77482001ef1c84e3333ed27be1b84ef8e01626bc45887d7a4de3fde1db3aa" Sep 05 08:03:04.460456 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.460380 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a77482001ef1c84e3333ed27be1b84ef8e01626bc45887d7a4de3fde1db3aa"} err="failed to get container status \"c4a77482001ef1c84e3333ed27be1b84ef8e01626bc45887d7a4de3fde1db3aa\": rpc error: code = NotFound desc = could not find container \"c4a77482001ef1c84e3333ed27be1b84ef8e01626bc45887d7a4de3fde1db3aa\": container with ID starting with c4a77482001ef1c84e3333ed27be1b84ef8e01626bc45887d7a4de3fde1db3aa not found: ID does not exist" Sep 05 08:03:04.460456 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.460403 2576 scope.go:117] "RemoveContainer" containerID="b417b4be96cc5e3064f59b19d1ac9a4faf32d6bd879210f094fd0f67fcb7862f" Sep 05 08:03:04.469089 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.469073 2576 scope.go:117] "RemoveContainer" containerID="b417b4be96cc5e3064f59b19d1ac9a4faf32d6bd879210f094fd0f67fcb7862f" Sep 05 08:03:04.469370 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:03:04.469352 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b417b4be96cc5e3064f59b19d1ac9a4faf32d6bd879210f094fd0f67fcb7862f\": container with ID starting with b417b4be96cc5e3064f59b19d1ac9a4faf32d6bd879210f094fd0f67fcb7862f not found: ID does not exist" containerID="b417b4be96cc5e3064f59b19d1ac9a4faf32d6bd879210f094fd0f67fcb7862f" Sep 05 08:03:04.469429 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.469376 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b417b4be96cc5e3064f59b19d1ac9a4faf32d6bd879210f094fd0f67fcb7862f"} err="failed to get container status \"b417b4be96cc5e3064f59b19d1ac9a4faf32d6bd879210f094fd0f67fcb7862f\": rpc error: code = NotFound desc = could not find container \"b417b4be96cc5e3064f59b19d1ac9a4faf32d6bd879210f094fd0f67fcb7862f\": container with ID starting with b417b4be96cc5e3064f59b19d1ac9a4faf32d6bd879210f094fd0f67fcb7862f not found: ID does not exist" Sep 05 08:03:04.477483 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.477458 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p"] Sep 05 08:03:04.482138 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.482118 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-4faf97-865f757746-wqw7p"] Sep 05 08:03:04.493403 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.493385 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p"] Sep 05 08:03:04.500581 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:04.500560 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-4faf97-5c77597d99-df92p"] Sep 05 08:03:05.886810 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:05.886767 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22270ab3-14df-4da1-be90-ad5d48d59570" path="/var/lib/kubelet/pods/22270ab3-14df-4da1-be90-ad5d48d59570/volumes" Sep 05 08:03:05.887362 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:05.887279 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c" path="/var/lib/kubelet/pods/d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c/volumes" Sep 05 08:03:07.454925 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:07.454887 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:03:17.902616 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:17.902585 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 08:03:17.903332 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:17.903308 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 08:03:17.906568 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:17.906545 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 08:03:17.907132 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:17.907111 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 08:03:20.438256 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.438219 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh"] Sep 05 08:03:20.438735 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.438436 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" podUID="9d6efcb5-8ec1-4635-91b9-b6573a869a47" containerName="payload-processing" containerID="cri-o://e5e1ddaa339490851d35b4e70b61ecac4030c6505fd3cdf977c9acf4e338357e" gracePeriod=30 Sep 05 08:03:20.445768 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.445746 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn"] Sep 05 08:03:20.445987 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.445960 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" podUID="d2a89ca1-eb15-4e5d-9c48-9772bd709620" containerName="payload-pre-processing" containerID="cri-o://217ea1b99754e20be6856543ccb1d277cb3b378bac27f39a49e497831ceee1c6" gracePeriod=30 Sep 05 08:03:20.714014 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.713986 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" Sep 05 08:03:20.732159 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.732129 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" Sep 05 08:03:20.815471 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.815441 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a89ca1-eb15-4e5d-9c48-9772bd709620-plugins-config-volume\") pod \"d2a89ca1-eb15-4e5d-9c48-9772bd709620\" (UID: \"d2a89ca1-eb15-4e5d-9c48-9772bd709620\") " Sep 05 08:03:20.815649 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.815490 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5gn9\" (UniqueName: \"kubernetes.io/projected/d2a89ca1-eb15-4e5d-9c48-9772bd709620-kube-api-access-l5gn9\") pod \"d2a89ca1-eb15-4e5d-9c48-9772bd709620\" (UID: \"d2a89ca1-eb15-4e5d-9c48-9772bd709620\") " Sep 05 08:03:20.815649 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.815531 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4gzp\" (UniqueName: \"kubernetes.io/projected/9d6efcb5-8ec1-4635-91b9-b6573a869a47-kube-api-access-j4gzp\") pod \"9d6efcb5-8ec1-4635-91b9-b6573a869a47\" (UID: \"9d6efcb5-8ec1-4635-91b9-b6573a869a47\") " Sep 05 08:03:20.815649 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.815599 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6efcb5-8ec1-4635-91b9-b6573a869a47-plugins-config-volume\") pod \"9d6efcb5-8ec1-4635-91b9-b6573a869a47\" (UID: \"9d6efcb5-8ec1-4635-91b9-b6573a869a47\") " Sep 05 08:03:20.815961 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.815929 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a89ca1-eb15-4e5d-9c48-9772bd709620-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2a89ca1-eb15-4e5d-9c48-9772bd709620" (UID: "d2a89ca1-eb15-4e5d-9c48-9772bd709620"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:03:20.816091 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.816048 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d6efcb5-8ec1-4635-91b9-b6573a869a47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d6efcb5-8ec1-4635-91b9-b6573a869a47" (UID: "9d6efcb5-8ec1-4635-91b9-b6573a869a47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:03:20.817991 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.817967 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d6efcb5-8ec1-4635-91b9-b6573a869a47-kube-api-access-j4gzp" (OuterVolumeSpecName: "kube-api-access-j4gzp") pod "9d6efcb5-8ec1-4635-91b9-b6573a869a47" (UID: "9d6efcb5-8ec1-4635-91b9-b6573a869a47"). InnerVolumeSpecName "kube-api-access-j4gzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:03:20.818097 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.817986 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a89ca1-eb15-4e5d-9c48-9772bd709620-kube-api-access-l5gn9" (OuterVolumeSpecName: "kube-api-access-l5gn9") pod "d2a89ca1-eb15-4e5d-9c48-9772bd709620" (UID: "d2a89ca1-eb15-4e5d-9c48-9772bd709620"). InnerVolumeSpecName "kube-api-access-l5gn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:03:20.916821 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.916785 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d6efcb5-8ec1-4635-91b9-b6573a869a47-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:03:20.916821 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.916813 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2a89ca1-eb15-4e5d-9c48-9772bd709620-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:03:20.916821 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.916823 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5gn9\" (UniqueName: \"kubernetes.io/projected/d2a89ca1-eb15-4e5d-9c48-9772bd709620-kube-api-access-l5gn9\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:03:20.916821 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:20.916832 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4gzp\" (UniqueName: \"kubernetes.io/projected/9d6efcb5-8ec1-4635-91b9-b6573a869a47-kube-api-access-j4gzp\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:03:21.546453 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.546418 2576 generic.go:356] "Generic (PLEG): container finished" podID="d2a89ca1-eb15-4e5d-9c48-9772bd709620" containerID="217ea1b99754e20be6856543ccb1d277cb3b378bac27f39a49e497831ceee1c6" exitCode=0 Sep 05 08:03:21.546948 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.546464 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" Sep 05 08:03:21.546948 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.546503 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" event={"ID":"d2a89ca1-eb15-4e5d-9c48-9772bd709620","Type":"ContainerDied","Data":"217ea1b99754e20be6856543ccb1d277cb3b378bac27f39a49e497831ceee1c6"} Sep 05 08:03:21.546948 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.546567 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn" event={"ID":"d2a89ca1-eb15-4e5d-9c48-9772bd709620","Type":"ContainerDied","Data":"d8ffe504eb4fb4e177908425b13d95d6390c931e6db400c0c0de2fff0f959e9c"} Sep 05 08:03:21.546948 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.546603 2576 scope.go:117] "RemoveContainer" containerID="217ea1b99754e20be6856543ccb1d277cb3b378bac27f39a49e497831ceee1c6" Sep 05 08:03:21.548098 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.548076 2576 generic.go:356] "Generic (PLEG): container finished" podID="9d6efcb5-8ec1-4635-91b9-b6573a869a47" containerID="e5e1ddaa339490851d35b4e70b61ecac4030c6505fd3cdf977c9acf4e338357e" exitCode=0 Sep 05 08:03:21.548167 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.548135 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" Sep 05 08:03:21.548167 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.548146 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" event={"ID":"9d6efcb5-8ec1-4635-91b9-b6573a869a47","Type":"ContainerDied","Data":"e5e1ddaa339490851d35b4e70b61ecac4030c6505fd3cdf977c9acf4e338357e"} Sep 05 08:03:21.548266 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.548182 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh" event={"ID":"9d6efcb5-8ec1-4635-91b9-b6573a869a47","Type":"ContainerDied","Data":"0f192fc5c69592a46cb3e8fa9b2e3544ae92b6de5ce59bc1b34f590346125b8c"} Sep 05 08:03:21.556548 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.556522 2576 scope.go:117] "RemoveContainer" containerID="217ea1b99754e20be6856543ccb1d277cb3b378bac27f39a49e497831ceee1c6" Sep 05 08:03:21.557051 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:03:21.556884 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217ea1b99754e20be6856543ccb1d277cb3b378bac27f39a49e497831ceee1c6\": container with ID starting with 217ea1b99754e20be6856543ccb1d277cb3b378bac27f39a49e497831ceee1c6 not found: ID does not exist" containerID="217ea1b99754e20be6856543ccb1d277cb3b378bac27f39a49e497831ceee1c6" Sep 05 08:03:21.557051 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.556916 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217ea1b99754e20be6856543ccb1d277cb3b378bac27f39a49e497831ceee1c6"} err="failed to get container status \"217ea1b99754e20be6856543ccb1d277cb3b378bac27f39a49e497831ceee1c6\": rpc error: code = NotFound desc = could not find container \"217ea1b99754e20be6856543ccb1d277cb3b378bac27f39a49e497831ceee1c6\": container with ID starting with 217ea1b99754e20be6856543ccb1d277cb3b378bac27f39a49e497831ceee1c6 not found: ID does not exist" Sep 05 08:03:21.557051 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.556941 2576 scope.go:117] "RemoveContainer" containerID="e5e1ddaa339490851d35b4e70b61ecac4030c6505fd3cdf977c9acf4e338357e" Sep 05 08:03:21.566123 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.566107 2576 scope.go:117] "RemoveContainer" containerID="e5e1ddaa339490851d35b4e70b61ecac4030c6505fd3cdf977c9acf4e338357e" Sep 05 08:03:21.566408 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:03:21.566389 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e1ddaa339490851d35b4e70b61ecac4030c6505fd3cdf977c9acf4e338357e\": container with ID starting with e5e1ddaa339490851d35b4e70b61ecac4030c6505fd3cdf977c9acf4e338357e not found: ID does not exist" containerID="e5e1ddaa339490851d35b4e70b61ecac4030c6505fd3cdf977c9acf4e338357e" Sep 05 08:03:21.566462 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.566416 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e1ddaa339490851d35b4e70b61ecac4030c6505fd3cdf977c9acf4e338357e"} err="failed to get container status \"e5e1ddaa339490851d35b4e70b61ecac4030c6505fd3cdf977c9acf4e338357e\": rpc error: code = NotFound desc = could not find container \"e5e1ddaa339490851d35b4e70b61ecac4030c6505fd3cdf977c9acf4e338357e\": container with ID starting with e5e1ddaa339490851d35b4e70b61ecac4030c6505fd3cdf977c9acf4e338357e not found: ID does not exist" Sep 05 08:03:21.575974 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.575947 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn"] Sep 05 08:03:21.583769 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.583747 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-46865726-69df8d474-fh2hn"] Sep 05 08:03:21.598110 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.598083 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh"] Sep 05 08:03:21.608361 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.608332 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-46865726-dcb6f7d6f-zb5jh"] Sep 05 08:03:21.893098 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.893057 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d6efcb5-8ec1-4635-91b9-b6573a869a47" path="/var/lib/kubelet/pods/9d6efcb5-8ec1-4635-91b9-b6573a869a47/volumes" Sep 05 08:03:21.893610 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:21.893593 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2a89ca1-eb15-4e5d-9c48-9772bd709620" path="/var/lib/kubelet/pods/d2a89ca1-eb15-4e5d-9c48-9772bd709620/volumes" Sep 05 08:03:26.679277 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:03:26.679239 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:04:05.048374 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:05.048340 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:04:16.602061 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.602020 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp"] Sep 05 08:04:16.602495 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.602409 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d6efcb5-8ec1-4635-91b9-b6573a869a47" containerName="payload-processing" Sep 05 08:04:16.602495 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.602422 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d6efcb5-8ec1-4635-91b9-b6573a869a47" containerName="payload-processing" Sep 05 08:04:16.602495 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.602442 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22270ab3-14df-4da1-be90-ad5d48d59570" containerName="payload-pre-processing" Sep 05 08:04:16.602495 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.602448 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22270ab3-14df-4da1-be90-ad5d48d59570" containerName="payload-pre-processing" Sep 05 08:04:16.602495 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.602456 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2a89ca1-eb15-4e5d-9c48-9772bd709620" containerName="payload-pre-processing" Sep 05 08:04:16.602495 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.602462 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2a89ca1-eb15-4e5d-9c48-9772bd709620" containerName="payload-pre-processing" Sep 05 08:04:16.602495 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.602471 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c" containerName="payload-processing" Sep 05 08:04:16.602495 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.602478 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c" containerName="payload-processing" Sep 05 08:04:16.602778 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.602529 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22270ab3-14df-4da1-be90-ad5d48d59570" containerName="payload-pre-processing" Sep 05 08:04:16.602778 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.602539 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2a89ca1-eb15-4e5d-9c48-9772bd709620" containerName="payload-pre-processing" Sep 05 08:04:16.602778 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.602545 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d6efcb5-8ec1-4635-91b9-b6573a869a47" containerName="payload-processing" Sep 05 08:04:16.602778 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.602550 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d44d63b2-9f5f-4681-b7a6-0eb6e98c5f2c" containerName="payload-processing" Sep 05 08:04:16.605595 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.605579 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" Sep 05 08:04:16.608079 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.608053 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-48a4b5\"" Sep 05 08:04:16.608189 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.608089 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-48a4b5-dockercfg-klrw9\"" Sep 05 08:04:16.616270 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.616245 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp"] Sep 05 08:04:16.623432 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.623407 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv"] Sep 05 08:04:16.626862 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.626838 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" Sep 05 08:04:16.638784 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.638763 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv"] Sep 05 08:04:16.716226 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.716182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2qbk\" (UniqueName: \"kubernetes.io/projected/05dcc23e-4d74-4824-8e92-a56d3dd5f5df-kube-api-access-k2qbk\") pod \"payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp\" (UID: \"05dcc23e-4d74-4824-8e92-a56d3dd5f5df\") " pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" Sep 05 08:04:16.716435 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.716247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv\" (UID: \"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" Sep 05 08:04:16.716435 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.716354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7wpw\" (UniqueName: \"kubernetes.io/projected/afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8-kube-api-access-n7wpw\") pod \"payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv\" (UID: \"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" Sep 05 08:04:16.716435 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.716404 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05dcc23e-4d74-4824-8e92-a56d3dd5f5df-plugins-config-volume\") pod \"payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp\" (UID: \"05dcc23e-4d74-4824-8e92-a56d3dd5f5df\") " pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" Sep 05 08:04:16.817734 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.817673 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv\" (UID: \"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" Sep 05 08:04:16.817940 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.817789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7wpw\" (UniqueName: \"kubernetes.io/projected/afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8-kube-api-access-n7wpw\") pod \"payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv\" (UID: \"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" Sep 05 08:04:16.817940 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.817868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05dcc23e-4d74-4824-8e92-a56d3dd5f5df-plugins-config-volume\") pod \"payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp\" (UID: \"05dcc23e-4d74-4824-8e92-a56d3dd5f5df\") " pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" Sep 05 08:04:16.818075 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.818000 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2qbk\" (UniqueName: \"kubernetes.io/projected/05dcc23e-4d74-4824-8e92-a56d3dd5f5df-kube-api-access-k2qbk\") pod \"payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp\" (UID: \"05dcc23e-4d74-4824-8e92-a56d3dd5f5df\") " pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" Sep 05 08:04:16.818545 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.818520 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv\" (UID: \"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" Sep 05 08:04:16.818978 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.818955 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05dcc23e-4d74-4824-8e92-a56d3dd5f5df-plugins-config-volume\") pod \"payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp\" (UID: \"05dcc23e-4d74-4824-8e92-a56d3dd5f5df\") " pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" Sep 05 08:04:16.826344 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.826318 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7wpw\" (UniqueName: \"kubernetes.io/projected/afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8-kube-api-access-n7wpw\") pod \"payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv\" (UID: \"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" Sep 05 08:04:16.826443 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.826368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2qbk\" (UniqueName: \"kubernetes.io/projected/05dcc23e-4d74-4824-8e92-a56d3dd5f5df-kube-api-access-k2qbk\") pod \"payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp\" (UID: \"05dcc23e-4d74-4824-8e92-a56d3dd5f5df\") " pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" Sep 05 08:04:16.916551 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.916471 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" Sep 05 08:04:16.938703 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:16.938678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" Sep 05 08:04:17.195222 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:17.195182 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp"] Sep 05 08:04:17.196178 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:04:17.196151 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05dcc23e_4d74_4824_8e92_a56d3dd5f5df.slice/crio-e60160a151a9c967da345d9e1ba2f72fb0211bc8707bdbc184820ecad2f60e6d WatchSource:0}: Error finding container e60160a151a9c967da345d9e1ba2f72fb0211bc8707bdbc184820ecad2f60e6d: Status 404 returned error can't find the container with id e60160a151a9c967da345d9e1ba2f72fb0211bc8707bdbc184820ecad2f60e6d Sep 05 08:04:17.223502 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:17.223475 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv"] Sep 05 08:04:17.233267 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:04:17.233234 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe93b0a_85ab_4f77_8fb6_3a4c8d6669c8.slice/crio-bc04829ed3a6ad8812442660ef72e1195efca8fe67e20b7f6cd4906c5b8de0b8 WatchSource:0}: Error finding container bc04829ed3a6ad8812442660ef72e1195efca8fe67e20b7f6cd4906c5b8de0b8: Status 404 returned error can't find the container with id bc04829ed3a6ad8812442660ef72e1195efca8fe67e20b7f6cd4906c5b8de0b8 Sep 05 08:04:17.847729 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:17.847697 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" event={"ID":"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8","Type":"ContainerStarted","Data":"5a754b2b7dee384b111ace796992f65e30af63effcb38899e6d9cc52461b0e3e"} Sep 05 08:04:17.848170 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:17.847758 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" event={"ID":"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8","Type":"ContainerStarted","Data":"bc04829ed3a6ad8812442660ef72e1195efca8fe67e20b7f6cd4906c5b8de0b8"} Sep 05 08:04:17.848170 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:17.847879 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" Sep 05 08:04:17.849417 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:17.849396 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" event={"ID":"05dcc23e-4d74-4824-8e92-a56d3dd5f5df","Type":"ContainerStarted","Data":"761d9e5fec4dc1a499afed5eb4851cad980d558464ec9db2e9452148a68b0a98"} Sep 05 08:04:17.849543 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:17.849444 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" event={"ID":"05dcc23e-4d74-4824-8e92-a56d3dd5f5df","Type":"ContainerStarted","Data":"e60160a151a9c967da345d9e1ba2f72fb0211bc8707bdbc184820ecad2f60e6d"} Sep 05 08:04:17.849543 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:17.849470 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" Sep 05 08:04:17.866845 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:17.866804 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" podStartSLOduration=1.866791799 podStartE2EDuration="1.866791799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:04:17.86461074 +0000 UTC m=+960.473585862" watchObservedRunningTime="2026-09-05 08:04:17.866791799 +0000 UTC m=+960.475766921" Sep 05 08:04:17.886173 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:17.886131 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" podStartSLOduration=1.886118369 podStartE2EDuration="1.886118369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:04:17.882951223 +0000 UTC m=+960.491926351" watchObservedRunningTime="2026-09-05 08:04:17.886118369 +0000 UTC m=+960.495093492" Sep 05 08:04:25.653518 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:25.653485 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:04:28.857415 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:28.857385 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" Sep 05 08:04:28.857927 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:28.857558 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" Sep 05 08:04:39.068061 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:39.067986 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:04:57.857732 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:04:57.857688 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:05:03.722067 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.722030 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62"] Sep 05 08:05:03.726853 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.726834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" Sep 05 08:05:03.729567 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.729542 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-66998bbc\"" Sep 05 08:05:03.737065 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.737039 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62"] Sep 05 08:05:03.764195 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.764151 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98"] Sep 05 08:05:03.767970 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.767951 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" Sep 05 08:05:03.770428 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.770402 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-66998bbc-dockercfg-2gk45\"" Sep 05 08:05:03.783660 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.783631 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98"] Sep 05 08:05:03.815945 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.815911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6s7q\" (UniqueName: \"kubernetes.io/projected/550ec6ec-5b01-4ceb-9fcd-ec282d5df189-kube-api-access-t6s7q\") pod \"payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98\" (UID: \"550ec6ec-5b01-4ceb-9fcd-ec282d5df189\") " pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" Sep 05 08:05:03.815945 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.815948 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dfd8\" (UniqueName: \"kubernetes.io/projected/757815ff-c9f5-4456-a0ab-2c2b7dbd4213-kube-api-access-7dfd8\") pod \"payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62\" (UID: \"757815ff-c9f5-4456-a0ab-2c2b7dbd4213\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" Sep 05 08:05:03.816186 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.816064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550ec6ec-5b01-4ceb-9fcd-ec282d5df189-plugins-config-volume\") pod \"payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98\" (UID: \"550ec6ec-5b01-4ceb-9fcd-ec282d5df189\") " pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" Sep 05 08:05:03.816186 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.816114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/757815ff-c9f5-4456-a0ab-2c2b7dbd4213-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62\" (UID: \"757815ff-c9f5-4456-a0ab-2c2b7dbd4213\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" Sep 05 08:05:03.917305 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.917267 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6s7q\" (UniqueName: \"kubernetes.io/projected/550ec6ec-5b01-4ceb-9fcd-ec282d5df189-kube-api-access-t6s7q\") pod \"payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98\" (UID: \"550ec6ec-5b01-4ceb-9fcd-ec282d5df189\") " pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" Sep 05 08:05:03.917305 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.917304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dfd8\" (UniqueName: \"kubernetes.io/projected/757815ff-c9f5-4456-a0ab-2c2b7dbd4213-kube-api-access-7dfd8\") pod \"payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62\" (UID: \"757815ff-c9f5-4456-a0ab-2c2b7dbd4213\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" Sep 05 08:05:03.917598 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.917373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550ec6ec-5b01-4ceb-9fcd-ec282d5df189-plugins-config-volume\") pod \"payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98\" (UID: \"550ec6ec-5b01-4ceb-9fcd-ec282d5df189\") " pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" Sep 05 08:05:03.917598 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.917408 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/757815ff-c9f5-4456-a0ab-2c2b7dbd4213-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62\" (UID: \"757815ff-c9f5-4456-a0ab-2c2b7dbd4213\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" Sep 05 08:05:03.918119 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.918100 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/757815ff-c9f5-4456-a0ab-2c2b7dbd4213-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62\" (UID: \"757815ff-c9f5-4456-a0ab-2c2b7dbd4213\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" Sep 05 08:05:03.918156 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.918100 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550ec6ec-5b01-4ceb-9fcd-ec282d5df189-plugins-config-volume\") pod \"payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98\" (UID: \"550ec6ec-5b01-4ceb-9fcd-ec282d5df189\") " pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" Sep 05 08:05:03.925535 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.925504 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6s7q\" (UniqueName: \"kubernetes.io/projected/550ec6ec-5b01-4ceb-9fcd-ec282d5df189-kube-api-access-t6s7q\") pod \"payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98\" (UID: \"550ec6ec-5b01-4ceb-9fcd-ec282d5df189\") " pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" Sep 05 08:05:03.925656 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:03.925586 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dfd8\" (UniqueName: \"kubernetes.io/projected/757815ff-c9f5-4456-a0ab-2c2b7dbd4213-kube-api-access-7dfd8\") pod \"payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62\" (UID: \"757815ff-c9f5-4456-a0ab-2c2b7dbd4213\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" Sep 05 08:05:04.037910 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:04.037817 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" Sep 05 08:05:04.078340 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:04.078300 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" Sep 05 08:05:04.315297 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:04.315270 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62"] Sep 05 08:05:04.318011 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:05:04.317982 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757815ff_c9f5_4456_a0ab_2c2b7dbd4213.slice/crio-93d05461123af695a838cdd3097304106fc5ac27b347cf8afa34dc087ee03474 WatchSource:0}: Error finding container 93d05461123af695a838cdd3097304106fc5ac27b347cf8afa34dc087ee03474: Status 404 returned error can't find the container with id 93d05461123af695a838cdd3097304106fc5ac27b347cf8afa34dc087ee03474 Sep 05 08:05:04.366713 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:04.366689 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98"] Sep 05 08:05:04.371373 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:05:04.371343 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550ec6ec_5b01_4ceb_9fcd_ec282d5df189.slice/crio-404d32bfd0efc7340e7dbf67b27d42445ef374ad7d820f50ed8c55577a02e15b WatchSource:0}: Error finding container 404d32bfd0efc7340e7dbf67b27d42445ef374ad7d820f50ed8c55577a02e15b: Status 404 returned error can't find the container with id 404d32bfd0efc7340e7dbf67b27d42445ef374ad7d820f50ed8c55577a02e15b Sep 05 08:05:05.118279 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:05.118242 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" event={"ID":"757815ff-c9f5-4456-a0ab-2c2b7dbd4213","Type":"ContainerStarted","Data":"9ec70f3f248ec315d292bba304cd0a5f205a1d3549ef182e317dfac2fae5df74"} Sep 05 08:05:05.118776 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:05.118303 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" event={"ID":"757815ff-c9f5-4456-a0ab-2c2b7dbd4213","Type":"ContainerStarted","Data":"93d05461123af695a838cdd3097304106fc5ac27b347cf8afa34dc087ee03474"} Sep 05 08:05:05.118776 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:05.118446 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" Sep 05 08:05:05.120050 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:05.120024 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" event={"ID":"550ec6ec-5b01-4ceb-9fcd-ec282d5df189","Type":"ContainerStarted","Data":"9b382a3fb9bef7bd0e653a3075088989b767092a590023ecfcddaf292d4fcc5a"} Sep 05 08:05:05.120192 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:05.120066 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" event={"ID":"550ec6ec-5b01-4ceb-9fcd-ec282d5df189","Type":"ContainerStarted","Data":"404d32bfd0efc7340e7dbf67b27d42445ef374ad7d820f50ed8c55577a02e15b"} Sep 05 08:05:05.120192 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:05.120149 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" Sep 05 08:05:05.135602 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:05.135553 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" podStartSLOduration=2.135538241 podStartE2EDuration="2.135538241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:05:05.134539304 +0000 UTC m=+1007.743514429" watchObservedRunningTime="2026-09-05 08:05:05.135538241 +0000 UTC m=+1007.744513363" Sep 05 08:05:05.154520 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:05.154468 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" podStartSLOduration=2.154445731 podStartE2EDuration="2.154445731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:05:05.152621893 +0000 UTC m=+1007.761597011" watchObservedRunningTime="2026-09-05 08:05:05.154445731 +0000 UTC m=+1007.763420853" Sep 05 08:05:16.127408 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:16.127371 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" Sep 05 08:05:16.127821 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:16.127585 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" Sep 05 08:05:17.953301 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:17.953261 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:05:29.403659 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.403625 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp"] Sep 05 08:05:29.404088 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.403841 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" podUID="05dcc23e-4d74-4824-8e92-a56d3dd5f5df" containerName="payload-processing" containerID="cri-o://761d9e5fec4dc1a499afed5eb4851cad980d558464ec9db2e9452148a68b0a98" gracePeriod=30 Sep 05 08:05:29.415924 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.415562 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv"] Sep 05 08:05:29.428891 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.416092 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" podUID="afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8" containerName="payload-pre-processing" containerID="cri-o://5a754b2b7dee384b111ace796992f65e30af63effcb38899e6d9cc52461b0e3e" gracePeriod=30 Sep 05 08:05:29.707549 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.707285 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" Sep 05 08:05:29.734937 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.734910 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" Sep 05 08:05:29.851947 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.851915 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8-plugins-config-volume\") pod \"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8\" (UID: \"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8\") " Sep 05 08:05:29.852142 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.851957 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2qbk\" (UniqueName: \"kubernetes.io/projected/05dcc23e-4d74-4824-8e92-a56d3dd5f5df-kube-api-access-k2qbk\") pod \"05dcc23e-4d74-4824-8e92-a56d3dd5f5df\" (UID: \"05dcc23e-4d74-4824-8e92-a56d3dd5f5df\") " Sep 05 08:05:29.852142 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.852014 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7wpw\" (UniqueName: \"kubernetes.io/projected/afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8-kube-api-access-n7wpw\") pod \"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8\" (UID: \"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8\") " Sep 05 08:05:29.852142 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.852059 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05dcc23e-4d74-4824-8e92-a56d3dd5f5df-plugins-config-volume\") pod \"05dcc23e-4d74-4824-8e92-a56d3dd5f5df\" (UID: \"05dcc23e-4d74-4824-8e92-a56d3dd5f5df\") " Sep 05 08:05:29.852445 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.852413 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8" (UID: "afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:05:29.852549 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.852523 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05dcc23e-4d74-4824-8e92-a56d3dd5f5df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05dcc23e-4d74-4824-8e92-a56d3dd5f5df" (UID: "05dcc23e-4d74-4824-8e92-a56d3dd5f5df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:05:29.854296 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.854274 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05dcc23e-4d74-4824-8e92-a56d3dd5f5df-kube-api-access-k2qbk" (OuterVolumeSpecName: "kube-api-access-k2qbk") pod "05dcc23e-4d74-4824-8e92-a56d3dd5f5df" (UID: "05dcc23e-4d74-4824-8e92-a56d3dd5f5df"). InnerVolumeSpecName "kube-api-access-k2qbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:05:29.854373 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.854354 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8-kube-api-access-n7wpw" (OuterVolumeSpecName: "kube-api-access-n7wpw") pod "afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8" (UID: "afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8"). InnerVolumeSpecName "kube-api-access-n7wpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:05:29.953599 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.953518 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7wpw\" (UniqueName: \"kubernetes.io/projected/afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8-kube-api-access-n7wpw\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:29.953599 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.953545 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05dcc23e-4d74-4824-8e92-a56d3dd5f5df-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:29.953599 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.953554 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:29.953599 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:29.953564 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2qbk\" (UniqueName: \"kubernetes.io/projected/05dcc23e-4d74-4824-8e92-a56d3dd5f5df-kube-api-access-k2qbk\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:30.262360 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.262272 2576 generic.go:356] "Generic (PLEG): container finished" podID="afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8" containerID="5a754b2b7dee384b111ace796992f65e30af63effcb38899e6d9cc52461b0e3e" exitCode=0 Sep 05 08:05:30.262360 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.262330 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" Sep 05 08:05:30.262590 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.262352 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" event={"ID":"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8","Type":"ContainerDied","Data":"5a754b2b7dee384b111ace796992f65e30af63effcb38899e6d9cc52461b0e3e"} Sep 05 08:05:30.262590 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.262447 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv" event={"ID":"afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8","Type":"ContainerDied","Data":"bc04829ed3a6ad8812442660ef72e1195efca8fe67e20b7f6cd4906c5b8de0b8"} Sep 05 08:05:30.262590 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.262487 2576 scope.go:117] "RemoveContainer" containerID="5a754b2b7dee384b111ace796992f65e30af63effcb38899e6d9cc52461b0e3e" Sep 05 08:05:30.263804 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.263777 2576 generic.go:356] "Generic (PLEG): container finished" podID="05dcc23e-4d74-4824-8e92-a56d3dd5f5df" containerID="761d9e5fec4dc1a499afed5eb4851cad980d558464ec9db2e9452148a68b0a98" exitCode=0 Sep 05 08:05:30.263933 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.263841 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" Sep 05 08:05:30.263933 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.263862 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" event={"ID":"05dcc23e-4d74-4824-8e92-a56d3dd5f5df","Type":"ContainerDied","Data":"761d9e5fec4dc1a499afed5eb4851cad980d558464ec9db2e9452148a68b0a98"} Sep 05 08:05:30.263933 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.263910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp" event={"ID":"05dcc23e-4d74-4824-8e92-a56d3dd5f5df","Type":"ContainerDied","Data":"e60160a151a9c967da345d9e1ba2f72fb0211bc8707bdbc184820ecad2f60e6d"} Sep 05 08:05:30.272080 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.272058 2576 scope.go:117] "RemoveContainer" containerID="5a754b2b7dee384b111ace796992f65e30af63effcb38899e6d9cc52461b0e3e" Sep 05 08:05:30.272375 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:05:30.272357 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a754b2b7dee384b111ace796992f65e30af63effcb38899e6d9cc52461b0e3e\": container with ID starting with 5a754b2b7dee384b111ace796992f65e30af63effcb38899e6d9cc52461b0e3e not found: ID does not exist" containerID="5a754b2b7dee384b111ace796992f65e30af63effcb38899e6d9cc52461b0e3e" Sep 05 08:05:30.272445 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.272386 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a754b2b7dee384b111ace796992f65e30af63effcb38899e6d9cc52461b0e3e"} err="failed to get container status \"5a754b2b7dee384b111ace796992f65e30af63effcb38899e6d9cc52461b0e3e\": rpc error: code = NotFound desc = could not find container \"5a754b2b7dee384b111ace796992f65e30af63effcb38899e6d9cc52461b0e3e\": container with ID starting with 5a754b2b7dee384b111ace796992f65e30af63effcb38899e6d9cc52461b0e3e not found: ID does not exist" Sep 05 08:05:30.272445 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.272405 2576 scope.go:117] "RemoveContainer" containerID="761d9e5fec4dc1a499afed5eb4851cad980d558464ec9db2e9452148a68b0a98" Sep 05 08:05:30.281089 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.281071 2576 scope.go:117] "RemoveContainer" containerID="761d9e5fec4dc1a499afed5eb4851cad980d558464ec9db2e9452148a68b0a98" Sep 05 08:05:30.281426 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:05:30.281407 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761d9e5fec4dc1a499afed5eb4851cad980d558464ec9db2e9452148a68b0a98\": container with ID starting with 761d9e5fec4dc1a499afed5eb4851cad980d558464ec9db2e9452148a68b0a98 not found: ID does not exist" containerID="761d9e5fec4dc1a499afed5eb4851cad980d558464ec9db2e9452148a68b0a98" Sep 05 08:05:30.281521 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.281436 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761d9e5fec4dc1a499afed5eb4851cad980d558464ec9db2e9452148a68b0a98"} err="failed to get container status \"761d9e5fec4dc1a499afed5eb4851cad980d558464ec9db2e9452148a68b0a98\": rpc error: code = NotFound desc = could not find container \"761d9e5fec4dc1a499afed5eb4851cad980d558464ec9db2e9452148a68b0a98\": container with ID starting with 761d9e5fec4dc1a499afed5eb4851cad980d558464ec9db2e9452148a68b0a98 not found: ID does not exist" Sep 05 08:05:30.283274 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.283254 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp"] Sep 05 08:05:30.287229 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.287193 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-48a4b5-74698f9fc7-tkgjp"] Sep 05 08:05:30.297676 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.297653 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv"] Sep 05 08:05:30.302960 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:30.302939 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-48a4b5-6f8b8ccc4-sx9mv"] Sep 05 08:05:31.886312 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:31.886278 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05dcc23e-4d74-4824-8e92-a56d3dd5f5df" path="/var/lib/kubelet/pods/05dcc23e-4d74-4824-8e92-a56d3dd5f5df/volumes" Sep 05 08:05:31.886693 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:31.886659 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8" path="/var/lib/kubelet/pods/afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8/volumes" Sep 05 08:05:32.056337 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:32.056300 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:05:44.958595 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:44.958562 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98"] Sep 05 08:05:44.959180 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:44.958827 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" podUID="550ec6ec-5b01-4ceb-9fcd-ec282d5df189" containerName="payload-processing" containerID="cri-o://9b382a3fb9bef7bd0e653a3075088989b767092a590023ecfcddaf292d4fcc5a" gracePeriod=30 Sep 05 08:05:44.965247 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:44.965201 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62"] Sep 05 08:05:44.965646 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:44.965619 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" podUID="757815ff-c9f5-4456-a0ab-2c2b7dbd4213" containerName="payload-pre-processing" containerID="cri-o://9ec70f3f248ec315d292bba304cd0a5f205a1d3549ef182e317dfac2fae5df74" gracePeriod=30 Sep 05 08:05:45.223464 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.223439 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" Sep 05 08:05:45.254303 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.254276 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" Sep 05 08:05:45.360839 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.360807 2576 generic.go:356] "Generic (PLEG): container finished" podID="550ec6ec-5b01-4ceb-9fcd-ec282d5df189" containerID="9b382a3fb9bef7bd0e653a3075088989b767092a590023ecfcddaf292d4fcc5a" exitCode=0 Sep 05 08:05:45.361029 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.360857 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" Sep 05 08:05:45.361029 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.360894 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" event={"ID":"550ec6ec-5b01-4ceb-9fcd-ec282d5df189","Type":"ContainerDied","Data":"9b382a3fb9bef7bd0e653a3075088989b767092a590023ecfcddaf292d4fcc5a"} Sep 05 08:05:45.361029 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.360951 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98" event={"ID":"550ec6ec-5b01-4ceb-9fcd-ec282d5df189","Type":"ContainerDied","Data":"404d32bfd0efc7340e7dbf67b27d42445ef374ad7d820f50ed8c55577a02e15b"} Sep 05 08:05:45.361029 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.360979 2576 scope.go:117] "RemoveContainer" containerID="9b382a3fb9bef7bd0e653a3075088989b767092a590023ecfcddaf292d4fcc5a" Sep 05 08:05:45.362366 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.362346 2576 generic.go:356] "Generic (PLEG): container finished" podID="757815ff-c9f5-4456-a0ab-2c2b7dbd4213" containerID="9ec70f3f248ec315d292bba304cd0a5f205a1d3549ef182e317dfac2fae5df74" exitCode=0 Sep 05 08:05:45.362454 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.362386 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" Sep 05 08:05:45.362454 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.362431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" event={"ID":"757815ff-c9f5-4456-a0ab-2c2b7dbd4213","Type":"ContainerDied","Data":"9ec70f3f248ec315d292bba304cd0a5f205a1d3549ef182e317dfac2fae5df74"} Sep 05 08:05:45.362528 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.362474 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62" event={"ID":"757815ff-c9f5-4456-a0ab-2c2b7dbd4213","Type":"ContainerDied","Data":"93d05461123af695a838cdd3097304106fc5ac27b347cf8afa34dc087ee03474"} Sep 05 08:05:45.370655 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.370634 2576 scope.go:117] "RemoveContainer" containerID="9b382a3fb9bef7bd0e653a3075088989b767092a590023ecfcddaf292d4fcc5a" Sep 05 08:05:45.370936 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:05:45.370919 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b382a3fb9bef7bd0e653a3075088989b767092a590023ecfcddaf292d4fcc5a\": container with ID starting with 9b382a3fb9bef7bd0e653a3075088989b767092a590023ecfcddaf292d4fcc5a not found: ID does not exist" containerID="9b382a3fb9bef7bd0e653a3075088989b767092a590023ecfcddaf292d4fcc5a" Sep 05 08:05:45.370992 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.370946 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b382a3fb9bef7bd0e653a3075088989b767092a590023ecfcddaf292d4fcc5a"} err="failed to get container status \"9b382a3fb9bef7bd0e653a3075088989b767092a590023ecfcddaf292d4fcc5a\": rpc error: code = NotFound desc = could not find container \"9b382a3fb9bef7bd0e653a3075088989b767092a590023ecfcddaf292d4fcc5a\": container with ID starting with 9b382a3fb9bef7bd0e653a3075088989b767092a590023ecfcddaf292d4fcc5a not found: ID does not exist" Sep 05 08:05:45.370992 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.370965 2576 scope.go:117] "RemoveContainer" containerID="9ec70f3f248ec315d292bba304cd0a5f205a1d3549ef182e317dfac2fae5df74" Sep 05 08:05:45.379149 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.379132 2576 scope.go:117] "RemoveContainer" containerID="9ec70f3f248ec315d292bba304cd0a5f205a1d3549ef182e317dfac2fae5df74" Sep 05 08:05:45.379488 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:05:45.379469 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec70f3f248ec315d292bba304cd0a5f205a1d3549ef182e317dfac2fae5df74\": container with ID starting with 9ec70f3f248ec315d292bba304cd0a5f205a1d3549ef182e317dfac2fae5df74 not found: ID does not exist" containerID="9ec70f3f248ec315d292bba304cd0a5f205a1d3549ef182e317dfac2fae5df74" Sep 05 08:05:45.379582 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.379491 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec70f3f248ec315d292bba304cd0a5f205a1d3549ef182e317dfac2fae5df74"} err="failed to get container status \"9ec70f3f248ec315d292bba304cd0a5f205a1d3549ef182e317dfac2fae5df74\": rpc error: code = NotFound desc = could not find container \"9ec70f3f248ec315d292bba304cd0a5f205a1d3549ef182e317dfac2fae5df74\": container with ID starting with 9ec70f3f248ec315d292bba304cd0a5f205a1d3549ef182e317dfac2fae5df74 not found: ID does not exist" Sep 05 08:05:45.388780 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.388757 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/757815ff-c9f5-4456-a0ab-2c2b7dbd4213-plugins-config-volume\") pod \"757815ff-c9f5-4456-a0ab-2c2b7dbd4213\" (UID: \"757815ff-c9f5-4456-a0ab-2c2b7dbd4213\") " Sep 05 08:05:45.388886 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.388795 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6s7q\" (UniqueName: \"kubernetes.io/projected/550ec6ec-5b01-4ceb-9fcd-ec282d5df189-kube-api-access-t6s7q\") pod \"550ec6ec-5b01-4ceb-9fcd-ec282d5df189\" (UID: \"550ec6ec-5b01-4ceb-9fcd-ec282d5df189\") " Sep 05 08:05:45.388886 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.388870 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dfd8\" (UniqueName: \"kubernetes.io/projected/757815ff-c9f5-4456-a0ab-2c2b7dbd4213-kube-api-access-7dfd8\") pod \"757815ff-c9f5-4456-a0ab-2c2b7dbd4213\" (UID: \"757815ff-c9f5-4456-a0ab-2c2b7dbd4213\") " Sep 05 08:05:45.388993 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.388906 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550ec6ec-5b01-4ceb-9fcd-ec282d5df189-plugins-config-volume\") pod \"550ec6ec-5b01-4ceb-9fcd-ec282d5df189\" (UID: \"550ec6ec-5b01-4ceb-9fcd-ec282d5df189\") " Sep 05 08:05:45.389282 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.389255 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757815ff-c9f5-4456-a0ab-2c2b7dbd4213-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "757815ff-c9f5-4456-a0ab-2c2b7dbd4213" (UID: "757815ff-c9f5-4456-a0ab-2c2b7dbd4213"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:05:45.389385 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.389306 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550ec6ec-5b01-4ceb-9fcd-ec282d5df189-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "550ec6ec-5b01-4ceb-9fcd-ec282d5df189" (UID: "550ec6ec-5b01-4ceb-9fcd-ec282d5df189"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:05:45.391106 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.391082 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550ec6ec-5b01-4ceb-9fcd-ec282d5df189-kube-api-access-t6s7q" (OuterVolumeSpecName: "kube-api-access-t6s7q") pod "550ec6ec-5b01-4ceb-9fcd-ec282d5df189" (UID: "550ec6ec-5b01-4ceb-9fcd-ec282d5df189"). InnerVolumeSpecName "kube-api-access-t6s7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:05:45.391200 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.391134 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757815ff-c9f5-4456-a0ab-2c2b7dbd4213-kube-api-access-7dfd8" (OuterVolumeSpecName: "kube-api-access-7dfd8") pod "757815ff-c9f5-4456-a0ab-2c2b7dbd4213" (UID: "757815ff-c9f5-4456-a0ab-2c2b7dbd4213"). InnerVolumeSpecName "kube-api-access-7dfd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:05:45.490056 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.489970 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/757815ff-c9f5-4456-a0ab-2c2b7dbd4213-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:45.490056 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.490006 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6s7q\" (UniqueName: \"kubernetes.io/projected/550ec6ec-5b01-4ceb-9fcd-ec282d5df189-kube-api-access-t6s7q\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:45.490056 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.490019 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dfd8\" (UniqueName: \"kubernetes.io/projected/757815ff-c9f5-4456-a0ab-2c2b7dbd4213-kube-api-access-7dfd8\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:45.490056 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.490036 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550ec6ec-5b01-4ceb-9fcd-ec282d5df189-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:45.697250 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.697175 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62"] Sep 05 08:05:45.701104 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.701076 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-66998bbc-78d8cc858b-scf62"] Sep 05 08:05:45.710158 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.710131 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98"] Sep 05 08:05:45.716562 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.716538 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-66998bbc-7fccb85889-4tv98"] Sep 05 08:05:45.887309 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.887278 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="550ec6ec-5b01-4ceb-9fcd-ec282d5df189" path="/var/lib/kubelet/pods/550ec6ec-5b01-4ceb-9fcd-ec282d5df189/volumes" Sep 05 08:05:45.887661 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:45.887645 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="757815ff-c9f5-4456-a0ab-2c2b7dbd4213" path="/var/lib/kubelet/pods/757815ff-c9f5-4456-a0ab-2c2b7dbd4213/volumes" Sep 05 08:05:49.647870 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:49.647827 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:05:51.044981 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.044944 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4"] Sep 05 08:05:51.045472 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.045176 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" podUID="17df67f4-2628-45ef-897c-0cb7345e11bc" containerName="maas-api" containerID="cri-o://1f530e3717f7dfa918c6939a74b71819bdc7849e6e639add69cc2de9f42c3d5a" gracePeriod=30 Sep 05 08:05:51.105259 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.105185 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd"] Sep 05 08:05:51.105848 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.105463 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" podUID="4b9656cd-5bac-4230-8526-8c21a0a96d05" containerName="payload-processing" containerID="cri-o://a0a464d1cd6be280efaf26782fc9db11a4a4bcd7e6839ee6c81bf057b9d5d072" gracePeriod=30 Sep 05 08:05:51.118702 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.118626 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5"] Sep 05 08:05:51.118997 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.118960 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" podUID="c618f1d4-a0bd-4786-b3da-34b3db49ab5a" containerName="payload-pre-processing" containerID="cri-o://31ac2c655a78ed0caa5cf2987348b98151799f7d9b3e25de304e3e682866dd7a" gracePeriod=30 Sep 05 08:05:51.411972 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.407340 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" Sep 05 08:05:51.411972 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.411939 2576 generic.go:356] "Generic (PLEG): container finished" podID="4b9656cd-5bac-4230-8526-8c21a0a96d05" containerID="a0a464d1cd6be280efaf26782fc9db11a4a4bcd7e6839ee6c81bf057b9d5d072" exitCode=0 Sep 05 08:05:51.412262 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.412196 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" event={"ID":"4b9656cd-5bac-4230-8526-8c21a0a96d05","Type":"ContainerDied","Data":"a0a464d1cd6be280efaf26782fc9db11a4a4bcd7e6839ee6c81bf057b9d5d072"} Sep 05 08:05:51.412333 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.412277 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" event={"ID":"4b9656cd-5bac-4230-8526-8c21a0a96d05","Type":"ContainerDied","Data":"480bcb710c38a8ef33b0f64fa95f552175c23252e8ba6cf6f774637389e8f916"} Sep 05 08:05:51.412333 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.412312 2576 scope.go:117] "RemoveContainer" containerID="a0a464d1cd6be280efaf26782fc9db11a4a4bcd7e6839ee6c81bf057b9d5d072" Sep 05 08:05:51.413997 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.412557 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd" Sep 05 08:05:51.418452 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.417173 2576 generic.go:356] "Generic (PLEG): container finished" podID="17df67f4-2628-45ef-897c-0cb7345e11bc" containerID="1f530e3717f7dfa918c6939a74b71819bdc7849e6e639add69cc2de9f42c3d5a" exitCode=0 Sep 05 08:05:51.418452 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.417363 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" event={"ID":"17df67f4-2628-45ef-897c-0cb7345e11bc","Type":"ContainerDied","Data":"1f530e3717f7dfa918c6939a74b71819bdc7849e6e639add69cc2de9f42c3d5a"} Sep 05 08:05:51.420126 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.420090 2576 generic.go:356] "Generic (PLEG): container finished" podID="c618f1d4-a0bd-4786-b3da-34b3db49ab5a" containerID="31ac2c655a78ed0caa5cf2987348b98151799f7d9b3e25de304e3e682866dd7a" exitCode=0 Sep 05 08:05:51.420263 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.420125 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" event={"ID":"c618f1d4-a0bd-4786-b3da-34b3db49ab5a","Type":"ContainerDied","Data":"31ac2c655a78ed0caa5cf2987348b98151799f7d9b3e25de304e3e682866dd7a"} Sep 05 08:05:51.426188 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.426166 2576 scope.go:117] "RemoveContainer" containerID="a0a464d1cd6be280efaf26782fc9db11a4a4bcd7e6839ee6c81bf057b9d5d072" Sep 05 08:05:51.426587 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:05:51.426566 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0a464d1cd6be280efaf26782fc9db11a4a4bcd7e6839ee6c81bf057b9d5d072\": container with ID starting with a0a464d1cd6be280efaf26782fc9db11a4a4bcd7e6839ee6c81bf057b9d5d072 not found: ID does not exist" containerID="a0a464d1cd6be280efaf26782fc9db11a4a4bcd7e6839ee6c81bf057b9d5d072" Sep 05 08:05:51.426680 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.426602 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0a464d1cd6be280efaf26782fc9db11a4a4bcd7e6839ee6c81bf057b9d5d072"} err="failed to get container status \"a0a464d1cd6be280efaf26782fc9db11a4a4bcd7e6839ee6c81bf057b9d5d072\": rpc error: code = NotFound desc = could not find container \"a0a464d1cd6be280efaf26782fc9db11a4a4bcd7e6839ee6c81bf057b9d5d072\": container with ID starting with a0a464d1cd6be280efaf26782fc9db11a4a4bcd7e6839ee6c81bf057b9d5d072 not found: ID does not exist" Sep 05 08:05:51.434333 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.434313 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" Sep 05 08:05:51.437764 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.437742 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjnsl\" (UniqueName: \"kubernetes.io/projected/c618f1d4-a0bd-4786-b3da-34b3db49ab5a-kube-api-access-mjnsl\") pod \"c618f1d4-a0bd-4786-b3da-34b3db49ab5a\" (UID: \"c618f1d4-a0bd-4786-b3da-34b3db49ab5a\") " Sep 05 08:05:51.437940 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.437778 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m8qq\" (UniqueName: \"kubernetes.io/projected/4b9656cd-5bac-4230-8526-8c21a0a96d05-kube-api-access-2m8qq\") pod \"4b9656cd-5bac-4230-8526-8c21a0a96d05\" (UID: \"4b9656cd-5bac-4230-8526-8c21a0a96d05\") " Sep 05 08:05:51.437940 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.437813 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c618f1d4-a0bd-4786-b3da-34b3db49ab5a-plugins-config-volume\") pod \"c618f1d4-a0bd-4786-b3da-34b3db49ab5a\" (UID: \"c618f1d4-a0bd-4786-b3da-34b3db49ab5a\") " Sep 05 08:05:51.437940 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.437842 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9656cd-5bac-4230-8526-8c21a0a96d05-plugins-config-volume\") pod \"4b9656cd-5bac-4230-8526-8c21a0a96d05\" (UID: \"4b9656cd-5bac-4230-8526-8c21a0a96d05\") " Sep 05 08:05:51.438268 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.438241 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9656cd-5bac-4230-8526-8c21a0a96d05-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b9656cd-5bac-4230-8526-8c21a0a96d05" (UID: "4b9656cd-5bac-4230-8526-8c21a0a96d05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:05:51.438630 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.438577 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c618f1d4-a0bd-4786-b3da-34b3db49ab5a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c618f1d4-a0bd-4786-b3da-34b3db49ab5a" (UID: "c618f1d4-a0bd-4786-b3da-34b3db49ab5a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:05:51.440717 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.440686 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9656cd-5bac-4230-8526-8c21a0a96d05-kube-api-access-2m8qq" (OuterVolumeSpecName: "kube-api-access-2m8qq") pod "4b9656cd-5bac-4230-8526-8c21a0a96d05" (UID: "4b9656cd-5bac-4230-8526-8c21a0a96d05"). InnerVolumeSpecName "kube-api-access-2m8qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:05:51.440913 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.440881 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c618f1d4-a0bd-4786-b3da-34b3db49ab5a-kube-api-access-mjnsl" (OuterVolumeSpecName: "kube-api-access-mjnsl") pod "c618f1d4-a0bd-4786-b3da-34b3db49ab5a" (UID: "c618f1d4-a0bd-4786-b3da-34b3db49ab5a"). InnerVolumeSpecName "kube-api-access-mjnsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:05:51.443456 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.443437 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:05:51.538896 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.538861 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-metrics-certs\") pod \"17df67f4-2628-45ef-897c-0cb7345e11bc\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " Sep 05 08:05:51.539065 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.538944 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pckk5\" (UniqueName: \"kubernetes.io/projected/17df67f4-2628-45ef-897c-0cb7345e11bc-kube-api-access-pckk5\") pod \"17df67f4-2628-45ef-897c-0cb7345e11bc\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " Sep 05 08:05:51.539065 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.538974 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17df67f4-2628-45ef-897c-0cb7345e11bc-openshift-service-ca\") pod \"17df67f4-2628-45ef-897c-0cb7345e11bc\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " Sep 05 08:05:51.539065 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.539051 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-maas-api-tls\") pod \"17df67f4-2628-45ef-897c-0cb7345e11bc\" (UID: \"17df67f4-2628-45ef-897c-0cb7345e11bc\") " Sep 05 08:05:51.539340 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.539314 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjnsl\" (UniqueName: \"kubernetes.io/projected/c618f1d4-a0bd-4786-b3da-34b3db49ab5a-kube-api-access-mjnsl\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:51.539340 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.539341 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m8qq\" (UniqueName: \"kubernetes.io/projected/4b9656cd-5bac-4230-8526-8c21a0a96d05-kube-api-access-2m8qq\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:51.539511 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.539355 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c618f1d4-a0bd-4786-b3da-34b3db49ab5a-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:51.539511 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.539369 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9656cd-5bac-4230-8526-8c21a0a96d05-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:51.539511 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.539423 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17df67f4-2628-45ef-897c-0cb7345e11bc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17df67f4-2628-45ef-897c-0cb7345e11bc" (UID: "17df67f4-2628-45ef-897c-0cb7345e11bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:05:51.541323 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.541293 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17df67f4-2628-45ef-897c-0cb7345e11bc-kube-api-access-pckk5" (OuterVolumeSpecName: "kube-api-access-pckk5") pod "17df67f4-2628-45ef-897c-0cb7345e11bc" (UID: "17df67f4-2628-45ef-897c-0cb7345e11bc"). InnerVolumeSpecName "kube-api-access-pckk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:05:51.541414 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.541349 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "17df67f4-2628-45ef-897c-0cb7345e11bc" (UID: "17df67f4-2628-45ef-897c-0cb7345e11bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:05:51.541414 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.541367 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17df67f4-2628-45ef-897c-0cb7345e11bc" (UID: "17df67f4-2628-45ef-897c-0cb7345e11bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:05:51.639841 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.639786 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-maas-api-tls\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:51.639841 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.639833 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17df67f4-2628-45ef-897c-0cb7345e11bc-metrics-certs\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:51.639841 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.639846 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pckk5\" (UniqueName: \"kubernetes.io/projected/17df67f4-2628-45ef-897c-0cb7345e11bc-kube-api-access-pckk5\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:51.640097 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.639859 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17df67f4-2628-45ef-897c-0cb7345e11bc-openshift-service-ca\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:05:51.733923 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.733866 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd"] Sep 05 08:05:51.738781 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.738753 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-e7c0ff-6c9d7d5799-v84gd"] Sep 05 08:05:51.886587 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:51.886550 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b9656cd-5bac-4230-8526-8c21a0a96d05" path="/var/lib/kubelet/pods/4b9656cd-5bac-4230-8526-8c21a0a96d05/volumes" Sep 05 08:05:52.427503 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:52.427453 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" event={"ID":"c618f1d4-a0bd-4786-b3da-34b3db49ab5a","Type":"ContainerDied","Data":"fdd72275dd5c038fee0f15935b17807ba73c00c26319261b2b680b8610931755"} Sep 05 08:05:52.428001 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:52.427531 2576 scope.go:117] "RemoveContainer" containerID="31ac2c655a78ed0caa5cf2987348b98151799f7d9b3e25de304e3e682866dd7a" Sep 05 08:05:52.428001 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:52.427463 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5" Sep 05 08:05:52.431055 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:52.431033 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" Sep 05 08:05:52.431055 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:52.431041 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4" event={"ID":"17df67f4-2628-45ef-897c-0cb7345e11bc","Type":"ContainerDied","Data":"e7b7c0f28edaadfb4d8e42f8860ebbcc10a4cad15b32c95666669d2ae4825b3c"} Sep 05 08:05:52.438368 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:52.438352 2576 scope.go:117] "RemoveContainer" containerID="1f530e3717f7dfa918c6939a74b71819bdc7849e6e639add69cc2de9f42c3d5a" Sep 05 08:05:52.448289 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:52.448259 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5"] Sep 05 08:05:52.450949 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:52.450926 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-e7c0ff-5bb64874c4-ggmb5"] Sep 05 08:05:52.462250 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:52.462203 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4"] Sep 05 08:05:52.468236 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:52.468194 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e7c0ff-6df844d779-vk7l4"] Sep 05 08:05:53.885711 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:53.885678 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17df67f4-2628-45ef-897c-0cb7345e11bc" path="/var/lib/kubelet/pods/17df67f4-2628-45ef-897c-0cb7345e11bc/volumes" Sep 05 08:05:53.886140 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:05:53.886014 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c618f1d4-a0bd-4786-b3da-34b3db49ab5a" path="/var/lib/kubelet/pods/c618f1d4-a0bd-4786-b3da-34b3db49ab5a/volumes" Sep 05 08:06:01.650286 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:01.650247 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:06:09.049535 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.049452 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6"] Sep 05 08:06:09.049977 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.049683 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" podUID="6c9a112a-7ec9-48d6-869a-f51636111f78" containerName="maas-api" containerID="cri-o://cdcc027dc1feef36a3d143261b41bb5ee6fcc9dd252a2276bffb02f991584f8d" gracePeriod=30 Sep 05 08:06:09.107998 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.107965 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx"] Sep 05 08:06:09.108282 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.108256 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" podUID="276bcafa-917b-4f8f-941b-0ad4080811c9" containerName="payload-processing" containerID="cri-o://97a16875192e9ff69439380c475181d20108fcc51c40e34310cb5ffcdf777f32" gracePeriod=30 Sep 05 08:06:09.116320 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.116274 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv"] Sep 05 08:06:09.116531 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.116508 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" podUID="0d5ade3f-cf42-4f42-9607-e315ee08c76d" containerName="payload-pre-processing" containerID="cri-o://3b192f9c50fcd45dfc3f8e6629435827306255db7b1991d18c412610c0b659ef" gracePeriod=30 Sep 05 08:06:09.372798 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.372764 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:06:09.414068 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.414032 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" Sep 05 08:06:09.417810 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.417788 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" Sep 05 08:06:09.480170 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.480138 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9a112a-7ec9-48d6-869a-f51636111f78-metrics-certs\") pod \"6c9a112a-7ec9-48d6-869a-f51636111f78\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " Sep 05 08:06:09.480361 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.480187 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74z4z\" (UniqueName: \"kubernetes.io/projected/0d5ade3f-cf42-4f42-9607-e315ee08c76d-kube-api-access-74z4z\") pod \"0d5ade3f-cf42-4f42-9607-e315ee08c76d\" (UID: \"0d5ade3f-cf42-4f42-9607-e315ee08c76d\") " Sep 05 08:06:09.480361 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.480233 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6kg7\" (UniqueName: \"kubernetes.io/projected/6c9a112a-7ec9-48d6-869a-f51636111f78-kube-api-access-m6kg7\") pod \"6c9a112a-7ec9-48d6-869a-f51636111f78\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " Sep 05 08:06:09.480361 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.480249 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5ade3f-cf42-4f42-9607-e315ee08c76d-plugins-config-volume\") pod \"0d5ade3f-cf42-4f42-9607-e315ee08c76d\" (UID: \"0d5ade3f-cf42-4f42-9607-e315ee08c76d\") " Sep 05 08:06:09.480361 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.480273 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9a112a-7ec9-48d6-869a-f51636111f78-maas-api-tls\") pod \"6c9a112a-7ec9-48d6-869a-f51636111f78\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " Sep 05 08:06:09.480361 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.480305 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zr7z\" (UniqueName: \"kubernetes.io/projected/276bcafa-917b-4f8f-941b-0ad4080811c9-kube-api-access-6zr7z\") pod \"276bcafa-917b-4f8f-941b-0ad4080811c9\" (UID: \"276bcafa-917b-4f8f-941b-0ad4080811c9\") " Sep 05 08:06:09.480646 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.480453 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276bcafa-917b-4f8f-941b-0ad4080811c9-plugins-config-volume\") pod \"276bcafa-917b-4f8f-941b-0ad4080811c9\" (UID: \"276bcafa-917b-4f8f-941b-0ad4080811c9\") " Sep 05 08:06:09.480646 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.480516 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a112a-7ec9-48d6-869a-f51636111f78-openshift-service-ca\") pod \"6c9a112a-7ec9-48d6-869a-f51636111f78\" (UID: \"6c9a112a-7ec9-48d6-869a-f51636111f78\") " Sep 05 08:06:09.480765 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.480729 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d5ade3f-cf42-4f42-9607-e315ee08c76d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0d5ade3f-cf42-4f42-9607-e315ee08c76d" (UID: "0d5ade3f-cf42-4f42-9607-e315ee08c76d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:06:09.481245 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.480850 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d5ade3f-cf42-4f42-9607-e315ee08c76d-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:06:09.481245 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.481069 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276bcafa-917b-4f8f-941b-0ad4080811c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "276bcafa-917b-4f8f-941b-0ad4080811c9" (UID: "276bcafa-917b-4f8f-941b-0ad4080811c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:06:09.481245 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.481226 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9a112a-7ec9-48d6-869a-f51636111f78-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c9a112a-7ec9-48d6-869a-f51636111f78" (UID: "6c9a112a-7ec9-48d6-869a-f51636111f78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:06:09.482715 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.482685 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d5ade3f-cf42-4f42-9607-e315ee08c76d-kube-api-access-74z4z" (OuterVolumeSpecName: "kube-api-access-74z4z") pod "0d5ade3f-cf42-4f42-9607-e315ee08c76d" (UID: "0d5ade3f-cf42-4f42-9607-e315ee08c76d"). InnerVolumeSpecName "kube-api-access-74z4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:06:09.482874 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.482854 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9a112a-7ec9-48d6-869a-f51636111f78-kube-api-access-m6kg7" (OuterVolumeSpecName: "kube-api-access-m6kg7") pod "6c9a112a-7ec9-48d6-869a-f51636111f78" (UID: "6c9a112a-7ec9-48d6-869a-f51636111f78"). InnerVolumeSpecName "kube-api-access-m6kg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:06:09.482913 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.482854 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9a112a-7ec9-48d6-869a-f51636111f78-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6c9a112a-7ec9-48d6-869a-f51636111f78" (UID: "6c9a112a-7ec9-48d6-869a-f51636111f78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:06:09.483077 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.483056 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9a112a-7ec9-48d6-869a-f51636111f78-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c9a112a-7ec9-48d6-869a-f51636111f78" (UID: "6c9a112a-7ec9-48d6-869a-f51636111f78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:06:09.483077 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.483055 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276bcafa-917b-4f8f-941b-0ad4080811c9-kube-api-access-6zr7z" (OuterVolumeSpecName: "kube-api-access-6zr7z") pod "276bcafa-917b-4f8f-941b-0ad4080811c9" (UID: "276bcafa-917b-4f8f-941b-0ad4080811c9"). InnerVolumeSpecName "kube-api-access-6zr7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:06:09.530226 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.530174 2576 generic.go:356] "Generic (PLEG): container finished" podID="276bcafa-917b-4f8f-941b-0ad4080811c9" containerID="97a16875192e9ff69439380c475181d20108fcc51c40e34310cb5ffcdf777f32" exitCode=0 Sep 05 08:06:09.530419 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.530250 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" Sep 05 08:06:09.530419 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.530277 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" event={"ID":"276bcafa-917b-4f8f-941b-0ad4080811c9","Type":"ContainerDied","Data":"97a16875192e9ff69439380c475181d20108fcc51c40e34310cb5ffcdf777f32"} Sep 05 08:06:09.530419 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.530376 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx" event={"ID":"276bcafa-917b-4f8f-941b-0ad4080811c9","Type":"ContainerDied","Data":"26508abe15817c8713113f308438887edcd716f592516a4d6def2162b27a2770"} Sep 05 08:06:09.530419 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.530408 2576 scope.go:117] "RemoveContainer" containerID="97a16875192e9ff69439380c475181d20108fcc51c40e34310cb5ffcdf777f32" Sep 05 08:06:09.531975 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.531955 2576 generic.go:356] "Generic (PLEG): container finished" podID="6c9a112a-7ec9-48d6-869a-f51636111f78" containerID="cdcc027dc1feef36a3d143261b41bb5ee6fcc9dd252a2276bffb02f991584f8d" exitCode=0 Sep 05 08:06:09.532078 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.532004 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" Sep 05 08:06:09.532078 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.532012 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" event={"ID":"6c9a112a-7ec9-48d6-869a-f51636111f78","Type":"ContainerDied","Data":"cdcc027dc1feef36a3d143261b41bb5ee6fcc9dd252a2276bffb02f991584f8d"} Sep 05 08:06:09.532078 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.532059 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6" event={"ID":"6c9a112a-7ec9-48d6-869a-f51636111f78","Type":"ContainerDied","Data":"18fb4cf22a54ab34331795ef031c4e5c32b0069aae7bd1257048dc7b1bdc9be9"} Sep 05 08:06:09.533973 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.533944 2576 generic.go:356] "Generic (PLEG): container finished" podID="0d5ade3f-cf42-4f42-9607-e315ee08c76d" containerID="3b192f9c50fcd45dfc3f8e6629435827306255db7b1991d18c412610c0b659ef" exitCode=0 Sep 05 08:06:09.534152 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.534013 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" Sep 05 08:06:09.534152 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.534030 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" event={"ID":"0d5ade3f-cf42-4f42-9607-e315ee08c76d","Type":"ContainerDied","Data":"3b192f9c50fcd45dfc3f8e6629435827306255db7b1991d18c412610c0b659ef"} Sep 05 08:06:09.534152 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.534092 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv" event={"ID":"0d5ade3f-cf42-4f42-9607-e315ee08c76d","Type":"ContainerDied","Data":"bc1afc52f88605b509f7422c4cf2877118897a5f81566660d22e647f6a6f415c"} Sep 05 08:06:09.542771 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.542740 2576 scope.go:117] "RemoveContainer" containerID="97a16875192e9ff69439380c475181d20108fcc51c40e34310cb5ffcdf777f32" Sep 05 08:06:09.543133 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:06:09.543108 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a16875192e9ff69439380c475181d20108fcc51c40e34310cb5ffcdf777f32\": container with ID starting with 97a16875192e9ff69439380c475181d20108fcc51c40e34310cb5ffcdf777f32 not found: ID does not exist" containerID="97a16875192e9ff69439380c475181d20108fcc51c40e34310cb5ffcdf777f32" Sep 05 08:06:09.543243 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.543144 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a16875192e9ff69439380c475181d20108fcc51c40e34310cb5ffcdf777f32"} err="failed to get container status \"97a16875192e9ff69439380c475181d20108fcc51c40e34310cb5ffcdf777f32\": rpc error: code = NotFound desc = could not find container \"97a16875192e9ff69439380c475181d20108fcc51c40e34310cb5ffcdf777f32\": container with ID starting with 97a16875192e9ff69439380c475181d20108fcc51c40e34310cb5ffcdf777f32 not found: ID does not exist" Sep 05 08:06:09.543243 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.543170 2576 scope.go:117] "RemoveContainer" containerID="cdcc027dc1feef36a3d143261b41bb5ee6fcc9dd252a2276bffb02f991584f8d" Sep 05 08:06:09.553253 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.553230 2576 scope.go:117] "RemoveContainer" containerID="cdcc027dc1feef36a3d143261b41bb5ee6fcc9dd252a2276bffb02f991584f8d" Sep 05 08:06:09.553519 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:06:09.553500 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdcc027dc1feef36a3d143261b41bb5ee6fcc9dd252a2276bffb02f991584f8d\": container with ID starting with cdcc027dc1feef36a3d143261b41bb5ee6fcc9dd252a2276bffb02f991584f8d not found: ID does not exist" containerID="cdcc027dc1feef36a3d143261b41bb5ee6fcc9dd252a2276bffb02f991584f8d" Sep 05 08:06:09.553572 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.553526 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdcc027dc1feef36a3d143261b41bb5ee6fcc9dd252a2276bffb02f991584f8d"} err="failed to get container status \"cdcc027dc1feef36a3d143261b41bb5ee6fcc9dd252a2276bffb02f991584f8d\": rpc error: code = NotFound desc = could not find container \"cdcc027dc1feef36a3d143261b41bb5ee6fcc9dd252a2276bffb02f991584f8d\": container with ID starting with cdcc027dc1feef36a3d143261b41bb5ee6fcc9dd252a2276bffb02f991584f8d not found: ID does not exist" Sep 05 08:06:09.553572 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.553542 2576 scope.go:117] "RemoveContainer" containerID="3b192f9c50fcd45dfc3f8e6629435827306255db7b1991d18c412610c0b659ef" Sep 05 08:06:09.561796 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.561772 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx"] Sep 05 08:06:09.563258 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.563242 2576 scope.go:117] "RemoveContainer" containerID="3b192f9c50fcd45dfc3f8e6629435827306255db7b1991d18c412610c0b659ef" Sep 05 08:06:09.563558 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:06:09.563539 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b192f9c50fcd45dfc3f8e6629435827306255db7b1991d18c412610c0b659ef\": container with ID starting with 3b192f9c50fcd45dfc3f8e6629435827306255db7b1991d18c412610c0b659ef not found: ID does not exist" containerID="3b192f9c50fcd45dfc3f8e6629435827306255db7b1991d18c412610c0b659ef" Sep 05 08:06:09.563614 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.563565 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b192f9c50fcd45dfc3f8e6629435827306255db7b1991d18c412610c0b659ef"} err="failed to get container status \"3b192f9c50fcd45dfc3f8e6629435827306255db7b1991d18c412610c0b659ef\": rpc error: code = NotFound desc = could not find container \"3b192f9c50fcd45dfc3f8e6629435827306255db7b1991d18c412610c0b659ef\": container with ID starting with 3b192f9c50fcd45dfc3f8e6629435827306255db7b1991d18c412610c0b659ef not found: ID does not exist" Sep 05 08:06:09.564716 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.564698 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f3ac-77d5c57fc5-pvjfx"] Sep 05 08:06:09.576326 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.576304 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6"] Sep 05 08:06:09.580744 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.580722 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f3ac-7c8964c6df-nvtg6"] Sep 05 08:06:09.581781 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.581762 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zr7z\" (UniqueName: \"kubernetes.io/projected/276bcafa-917b-4f8f-941b-0ad4080811c9-kube-api-access-6zr7z\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:06:09.581826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.581784 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/276bcafa-917b-4f8f-941b-0ad4080811c9-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:06:09.581826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.581795 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9a112a-7ec9-48d6-869a-f51636111f78-openshift-service-ca\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:06:09.581826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.581805 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9a112a-7ec9-48d6-869a-f51636111f78-metrics-certs\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:06:09.581826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.581814 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74z4z\" (UniqueName: \"kubernetes.io/projected/0d5ade3f-cf42-4f42-9607-e315ee08c76d-kube-api-access-74z4z\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:06:09.581826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.581822 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6kg7\" (UniqueName: \"kubernetes.io/projected/6c9a112a-7ec9-48d6-869a-f51636111f78-kube-api-access-m6kg7\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:06:09.581989 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.581830 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9a112a-7ec9-48d6-869a-f51636111f78-maas-api-tls\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:06:09.595408 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.595385 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv"] Sep 05 08:06:09.600658 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.600638 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f3ac-75c99f6dfd-hgnvv"] Sep 05 08:06:09.886390 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.886356 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d5ade3f-cf42-4f42-9607-e315ee08c76d" path="/var/lib/kubelet/pods/0d5ade3f-cf42-4f42-9607-e315ee08c76d/volumes" Sep 05 08:06:09.886800 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.886781 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276bcafa-917b-4f8f-941b-0ad4080811c9" path="/var/lib/kubelet/pods/276bcafa-917b-4f8f-941b-0ad4080811c9/volumes" Sep 05 08:06:09.887179 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:09.887162 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c9a112a-7ec9-48d6-869a-f51636111f78" path="/var/lib/kubelet/pods/6c9a112a-7ec9-48d6-869a-f51636111f78/volumes" Sep 05 08:06:11.097115 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.097074 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:06:11.519118 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519041 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h"] Sep 05 08:06:11.519390 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519376 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b9656cd-5bac-4230-8526-8c21a0a96d05" containerName="payload-processing" Sep 05 08:06:11.519442 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519392 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b9656cd-5bac-4230-8526-8c21a0a96d05" containerName="payload-processing" Sep 05 08:06:11.519442 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519404 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17df67f4-2628-45ef-897c-0cb7345e11bc" containerName="maas-api" Sep 05 08:06:11.519442 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519410 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="17df67f4-2628-45ef-897c-0cb7345e11bc" containerName="maas-api" Sep 05 08:06:11.519442 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519416 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8" containerName="payload-pre-processing" Sep 05 08:06:11.519442 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519422 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8" containerName="payload-pre-processing" Sep 05 08:06:11.519442 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519434 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276bcafa-917b-4f8f-941b-0ad4080811c9" containerName="payload-processing" Sep 05 08:06:11.519442 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519438 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="276bcafa-917b-4f8f-941b-0ad4080811c9" containerName="payload-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519445 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d5ade3f-cf42-4f42-9607-e315ee08c76d" containerName="payload-pre-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519450 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d5ade3f-cf42-4f42-9607-e315ee08c76d" containerName="payload-pre-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519457 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="757815ff-c9f5-4456-a0ab-2c2b7dbd4213" containerName="payload-pre-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519463 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="757815ff-c9f5-4456-a0ab-2c2b7dbd4213" containerName="payload-pre-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519470 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c618f1d4-a0bd-4786-b3da-34b3db49ab5a" containerName="payload-pre-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519475 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c618f1d4-a0bd-4786-b3da-34b3db49ab5a" containerName="payload-pre-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519486 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c9a112a-7ec9-48d6-869a-f51636111f78" containerName="maas-api" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519492 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c9a112a-7ec9-48d6-869a-f51636111f78" containerName="maas-api" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519497 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05dcc23e-4d74-4824-8e92-a56d3dd5f5df" containerName="payload-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519512 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="05dcc23e-4d74-4824-8e92-a56d3dd5f5df" containerName="payload-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519521 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="550ec6ec-5b01-4ceb-9fcd-ec282d5df189" containerName="payload-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519525 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="550ec6ec-5b01-4ceb-9fcd-ec282d5df189" containerName="payload-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519570 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d5ade3f-cf42-4f42-9607-e315ee08c76d" containerName="payload-pre-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519577 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="550ec6ec-5b01-4ceb-9fcd-ec282d5df189" containerName="payload-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519584 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="757815ff-c9f5-4456-a0ab-2c2b7dbd4213" containerName="payload-pre-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519591 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c9a112a-7ec9-48d6-869a-f51636111f78" containerName="maas-api" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519596 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c618f1d4-a0bd-4786-b3da-34b3db49ab5a" containerName="payload-pre-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519601 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b9656cd-5bac-4230-8526-8c21a0a96d05" containerName="payload-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519607 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="276bcafa-917b-4f8f-941b-0ad4080811c9" containerName="payload-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519613 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="afe93b0a-85ab-4f77-8fb6-3a4c8d6669c8" containerName="payload-pre-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519618 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="05dcc23e-4d74-4824-8e92-a56d3dd5f5df" containerName="payload-processing" Sep 05 08:06:11.519654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.519622 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="17df67f4-2628-45ef-897c-0cb7345e11bc" containerName="maas-api" Sep 05 08:06:11.524224 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.524185 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:11.526536 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.526512 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 08:06:11.526690 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.526548 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-8b6928-serving-cert\"" Sep 05 08:06:11.526690 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.526651 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 05 08:06:11.526801 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.526748 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c62fz\"" Sep 05 08:06:11.527034 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.527015 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 05 08:06:11.533498 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.533411 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h"] Sep 05 08:06:11.598146 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.598114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:11.598350 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.598166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:11.598350 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.598193 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lkgx\" (UniqueName: \"kubernetes.io/projected/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-kube-api-access-5lkgx\") pod \"maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:11.598350 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.598316 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:11.699334 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.699293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lkgx\" (UniqueName: \"kubernetes.io/projected/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-kube-api-access-5lkgx\") pod \"maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:11.699558 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.699356 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:11.699558 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.699419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:11.699558 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.699469 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:11.700122 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.700095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:11.702105 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.702078 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:11.702237 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.702179 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:11.707638 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.707613 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lkgx\" (UniqueName: \"kubernetes.io/projected/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-kube-api-access-5lkgx\") pod \"maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:11.837291 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:11.837261 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:12.115545 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.115521 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h"] Sep 05 08:06:12.118323 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:06:12.118284 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b615b9_5ab3_4aeb_aac8_8f4cb35fb2eb.slice/crio-ed91a56c9f03d34b17dc0d1dc2a9cf9f1bf70934e536ff34617483a0b0820ab8 WatchSource:0}: Error finding container ed91a56c9f03d34b17dc0d1dc2a9cf9f1bf70934e536ff34617483a0b0820ab8: Status 404 returned error can't find the container with id ed91a56c9f03d34b17dc0d1dc2a9cf9f1bf70934e536ff34617483a0b0820ab8 Sep 05 08:06:12.121560 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.120115 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 08:06:12.153194 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.153164 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml"] Sep 05 08:06:12.162616 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.162583 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" Sep 05 08:06:12.165406 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.165381 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-8b6928\"" Sep 05 08:06:12.170019 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.169990 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml"] Sep 05 08:06:12.176515 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.176491 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx"] Sep 05 08:06:12.179641 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.179620 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" Sep 05 08:06:12.182303 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.182278 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-8b6928-dockercfg-jf8pd\"" Sep 05 08:06:12.191315 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.191289 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx"] Sep 05 08:06:12.203660 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.203631 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d592f3e0-53bd-4928-a47a-f53d54c6600d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx\" (UID: \"d592f3e0-53bd-4928-a47a-f53d54c6600d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" Sep 05 08:06:12.203799 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.203687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb0ed63c-a287-4d29-999f-0d6c9c950064-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml\" (UID: \"cb0ed63c-a287-4d29-999f-0d6c9c950064\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" Sep 05 08:06:12.203799 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.203708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s7m8\" (UniqueName: \"kubernetes.io/projected/cb0ed63c-a287-4d29-999f-0d6c9c950064-kube-api-access-7s7m8\") pod \"payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml\" (UID: \"cb0ed63c-a287-4d29-999f-0d6c9c950064\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" Sep 05 08:06:12.203799 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.203747 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjzg5\" (UniqueName: \"kubernetes.io/projected/d592f3e0-53bd-4928-a47a-f53d54c6600d-kube-api-access-zjzg5\") pod \"payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx\" (UID: \"d592f3e0-53bd-4928-a47a-f53d54c6600d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" Sep 05 08:06:12.304605 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.304572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjzg5\" (UniqueName: \"kubernetes.io/projected/d592f3e0-53bd-4928-a47a-f53d54c6600d-kube-api-access-zjzg5\") pod \"payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx\" (UID: \"d592f3e0-53bd-4928-a47a-f53d54c6600d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" Sep 05 08:06:12.304792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.304618 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d592f3e0-53bd-4928-a47a-f53d54c6600d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx\" (UID: \"d592f3e0-53bd-4928-a47a-f53d54c6600d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" Sep 05 08:06:12.304792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.304658 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb0ed63c-a287-4d29-999f-0d6c9c950064-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml\" (UID: \"cb0ed63c-a287-4d29-999f-0d6c9c950064\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" Sep 05 08:06:12.304792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.304678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s7m8\" (UniqueName: \"kubernetes.io/projected/cb0ed63c-a287-4d29-999f-0d6c9c950064-kube-api-access-7s7m8\") pod \"payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml\" (UID: \"cb0ed63c-a287-4d29-999f-0d6c9c950064\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" Sep 05 08:06:12.305404 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.305382 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb0ed63c-a287-4d29-999f-0d6c9c950064-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml\" (UID: \"cb0ed63c-a287-4d29-999f-0d6c9c950064\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" Sep 05 08:06:12.305503 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.305435 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d592f3e0-53bd-4928-a47a-f53d54c6600d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx\" (UID: \"d592f3e0-53bd-4928-a47a-f53d54c6600d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" Sep 05 08:06:12.312798 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.312771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s7m8\" (UniqueName: \"kubernetes.io/projected/cb0ed63c-a287-4d29-999f-0d6c9c950064-kube-api-access-7s7m8\") pod \"payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml\" (UID: \"cb0ed63c-a287-4d29-999f-0d6c9c950064\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" Sep 05 08:06:12.312891 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.312776 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjzg5\" (UniqueName: \"kubernetes.io/projected/d592f3e0-53bd-4928-a47a-f53d54c6600d-kube-api-access-zjzg5\") pod \"payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx\" (UID: \"d592f3e0-53bd-4928-a47a-f53d54c6600d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" Sep 05 08:06:12.476952 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.476850 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" Sep 05 08:06:12.491044 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.491019 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" Sep 05 08:06:12.556372 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.556282 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" event={"ID":"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb","Type":"ContainerStarted","Data":"ed91a56c9f03d34b17dc0d1dc2a9cf9f1bf70934e536ff34617483a0b0820ab8"} Sep 05 08:06:12.754158 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.754073 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml"] Sep 05 08:06:12.758085 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:06:12.758058 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0ed63c_a287_4d29_999f_0d6c9c950064.slice/crio-5f4efb974fe26e05afbbe9ed2189c0533e125567476fb7bd35efc244d425ec4b WatchSource:0}: Error finding container 5f4efb974fe26e05afbbe9ed2189c0533e125567476fb7bd35efc244d425ec4b: Status 404 returned error can't find the container with id 5f4efb974fe26e05afbbe9ed2189c0533e125567476fb7bd35efc244d425ec4b Sep 05 08:06:12.782716 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:12.782688 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx"] Sep 05 08:06:12.783679 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:06:12.783649 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd592f3e0_53bd_4928_a47a_f53d54c6600d.slice/crio-b079e00432c02dbefe8178505ff3ef652d9e695244a06649475e8e6d8844207c WatchSource:0}: Error finding container b079e00432c02dbefe8178505ff3ef652d9e695244a06649475e8e6d8844207c: Status 404 returned error can't find the container with id b079e00432c02dbefe8178505ff3ef652d9e695244a06649475e8e6d8844207c Sep 05 08:06:13.563816 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:13.563778 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" event={"ID":"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb","Type":"ContainerStarted","Data":"a2521b2b779edac51c360cd2370de15c960df70ff1c4d98c28267a947cb34e8e"} Sep 05 08:06:13.564410 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:13.563950 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:13.565428 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:13.565401 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" event={"ID":"d592f3e0-53bd-4928-a47a-f53d54c6600d","Type":"ContainerStarted","Data":"68d4b1a3f33824d21bc2bb208015aa3b5fec5c7c89ce45a5f42ff6cae28673f8"} Sep 05 08:06:13.565544 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:13.565444 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" event={"ID":"d592f3e0-53bd-4928-a47a-f53d54c6600d","Type":"ContainerStarted","Data":"b079e00432c02dbefe8178505ff3ef652d9e695244a06649475e8e6d8844207c"} Sep 05 08:06:13.565544 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:13.565470 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" Sep 05 08:06:13.566949 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:13.566928 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" event={"ID":"cb0ed63c-a287-4d29-999f-0d6c9c950064","Type":"ContainerStarted","Data":"e23ca6bce8562dcfb634b02b3e059d0cb87717068042e6a5c63be86b53a94921"} Sep 05 08:06:13.567052 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:13.566962 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" event={"ID":"cb0ed63c-a287-4d29-999f-0d6c9c950064","Type":"ContainerStarted","Data":"5f4efb974fe26e05afbbe9ed2189c0533e125567476fb7bd35efc244d425ec4b"} Sep 05 08:06:13.567052 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:13.566997 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" Sep 05 08:06:13.583000 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:13.582960 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" podStartSLOduration=2.108517356 podStartE2EDuration="2.582946244s" totalImagesPullingTime="474.428888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:06:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:06:13.580136386 +0000 UTC m=+1076.189111510" watchObservedRunningTime="2026-09-05 08:06:13.582946244 +0000 UTC m=+1076.191921357" Sep 05 08:06:13.596595 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:13.596546 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" podStartSLOduration=1.596531639 podStartE2EDuration="1.596531639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:06:13.59493924 +0000 UTC m=+1076.203914366" watchObservedRunningTime="2026-09-05 08:06:13.596531639 +0000 UTC m=+1076.205506773" Sep 05 08:06:13.611894 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:13.611845 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" podStartSLOduration=1.611830727 podStartE2EDuration="1.611830727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:06:13.6096292 +0000 UTC m=+1076.218604329" watchObservedRunningTime="2026-09-05 08:06:13.611830727 +0000 UTC m=+1076.220805914" Sep 05 08:06:19.578273 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:19.578229 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:06:20.547872 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:20.547835 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:06:21.612282 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.612171 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj"] Sep 05 08:06:21.618025 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.617998 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:21.621196 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.621172 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-8c2e20-serving-cert\"" Sep 05 08:06:21.631414 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.631379 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj"] Sep 05 08:06:21.688451 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.688404 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8m8\" (UniqueName: \"kubernetes.io/projected/ee57f33b-65e7-459d-93b9-76292560fe93-kube-api-access-zn8m8\") pod \"maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:21.688451 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.688469 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:21.688741 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.688598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee57f33b-65e7-459d-93b9-76292560fe93-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:21.688741 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.688649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:21.789488 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.789443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee57f33b-65e7-459d-93b9-76292560fe93-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:21.789700 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.789506 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:21.789700 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.789572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8m8\" (UniqueName: \"kubernetes.io/projected/ee57f33b-65e7-459d-93b9-76292560fe93-kube-api-access-zn8m8\") pod \"maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:21.789700 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.789620 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:21.789877 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:06:21.789755 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-serving-cert: secret "maas-api-e2e-shared-a-w3-8c2e20-serving-cert" not found Sep 05 08:06:21.789877 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:06:21.789822 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-maas-api-tls podName:ee57f33b-65e7-459d-93b9-76292560fe93 nodeName:}" failed. No retries permitted until 2026-09-05 08:06:22.289800511 +0000 UTC m=+1084.898775635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-maas-api-tls") pod "maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" (UID: "ee57f33b-65e7-459d-93b9-76292560fe93") : secret "maas-api-e2e-shared-a-w3-8c2e20-serving-cert" not found Sep 05 08:06:21.790273 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.790252 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee57f33b-65e7-459d-93b9-76292560fe93-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:21.792860 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.792821 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:21.800602 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:21.800580 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8m8\" (UniqueName: \"kubernetes.io/projected/ee57f33b-65e7-459d-93b9-76292560fe93-kube-api-access-zn8m8\") pod \"maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:22.294443 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.294403 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:22.296948 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.296919 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:22.409914 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.409859 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22"] Sep 05 08:06:22.413633 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.413610 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" Sep 05 08:06:22.416114 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.416082 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-8c2e20-dockercfg-mzhjc\"" Sep 05 08:06:22.416282 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.416094 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-8c2e20\"" Sep 05 08:06:22.425865 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.425839 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22"] Sep 05 08:06:22.481643 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.481608 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk"] Sep 05 08:06:22.485127 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.485110 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" Sep 05 08:06:22.496314 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.496280 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtgjn\" (UniqueName: \"kubernetes.io/projected/8872c321-9589-4d4e-b78f-1dcfa38b9462-kube-api-access-mtgjn\") pod \"payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22\" (UID: \"8872c321-9589-4d4e-b78f-1dcfa38b9462\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" Sep 05 08:06:22.496314 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.496322 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk"] Sep 05 08:06:22.496543 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.496370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8872c321-9589-4d4e-b78f-1dcfa38b9462-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22\" (UID: \"8872c321-9589-4d4e-b78f-1dcfa38b9462\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" Sep 05 08:06:22.530334 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.530306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:22.597880 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.597831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtgjn\" (UniqueName: \"kubernetes.io/projected/8872c321-9589-4d4e-b78f-1dcfa38b9462-kube-api-access-mtgjn\") pod \"payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22\" (UID: \"8872c321-9589-4d4e-b78f-1dcfa38b9462\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" Sep 05 08:06:22.598111 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.597941 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8872c321-9589-4d4e-b78f-1dcfa38b9462-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22\" (UID: \"8872c321-9589-4d4e-b78f-1dcfa38b9462\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" Sep 05 08:06:22.598111 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.597986 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpjpj\" (UniqueName: \"kubernetes.io/projected/bf0cb253-b79a-4906-8986-fcbdd291d8f9-kube-api-access-cpjpj\") pod \"payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk\" (UID: \"bf0cb253-b79a-4906-8986-fcbdd291d8f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" Sep 05 08:06:22.598111 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.598077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0cb253-b79a-4906-8986-fcbdd291d8f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk\" (UID: \"bf0cb253-b79a-4906-8986-fcbdd291d8f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" Sep 05 08:06:22.599406 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.599356 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8872c321-9589-4d4e-b78f-1dcfa38b9462-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22\" (UID: \"8872c321-9589-4d4e-b78f-1dcfa38b9462\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" Sep 05 08:06:22.606958 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.606936 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtgjn\" (UniqueName: \"kubernetes.io/projected/8872c321-9589-4d4e-b78f-1dcfa38b9462-kube-api-access-mtgjn\") pod \"payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22\" (UID: \"8872c321-9589-4d4e-b78f-1dcfa38b9462\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" Sep 05 08:06:22.699025 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.698987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpjpj\" (UniqueName: \"kubernetes.io/projected/bf0cb253-b79a-4906-8986-fcbdd291d8f9-kube-api-access-cpjpj\") pod \"payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk\" (UID: \"bf0cb253-b79a-4906-8986-fcbdd291d8f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" Sep 05 08:06:22.699565 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.699053 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0cb253-b79a-4906-8986-fcbdd291d8f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk\" (UID: \"bf0cb253-b79a-4906-8986-fcbdd291d8f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" Sep 05 08:06:22.699774 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.699753 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0cb253-b79a-4906-8986-fcbdd291d8f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk\" (UID: \"bf0cb253-b79a-4906-8986-fcbdd291d8f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" Sep 05 08:06:22.707479 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.707451 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpjpj\" (UniqueName: \"kubernetes.io/projected/bf0cb253-b79a-4906-8986-fcbdd291d8f9-kube-api-access-cpjpj\") pod \"payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk\" (UID: \"bf0cb253-b79a-4906-8986-fcbdd291d8f9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" Sep 05 08:06:22.725434 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.725411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" Sep 05 08:06:22.796105 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.796078 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" Sep 05 08:06:22.809438 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.809175 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj"] Sep 05 08:06:22.812165 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:06:22.812137 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee57f33b_65e7_459d_93b9_76292560fe93.slice/crio-a8a3009ec743f82ab6238863827085b5ba1fd1f1c3294cbf6017a06a4f12e78e WatchSource:0}: Error finding container a8a3009ec743f82ab6238863827085b5ba1fd1f1c3294cbf6017a06a4f12e78e: Status 404 returned error can't find the container with id a8a3009ec743f82ab6238863827085b5ba1fd1f1c3294cbf6017a06a4f12e78e Sep 05 08:06:22.998985 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:22.998960 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22"] Sep 05 08:06:23.000897 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:06:23.000869 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8872c321_9589_4d4e_b78f_1dcfa38b9462.slice/crio-2f99e4221a8031b3435ade933eb6995899604dc4d1508cc40447125b72763818 WatchSource:0}: Error finding container 2f99e4221a8031b3435ade933eb6995899604dc4d1508cc40447125b72763818: Status 404 returned error can't find the container with id 2f99e4221a8031b3435ade933eb6995899604dc4d1508cc40447125b72763818 Sep 05 08:06:23.081366 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:23.081336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk"] Sep 05 08:06:23.084164 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:06:23.084131 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0cb253_b79a_4906_8986_fcbdd291d8f9.slice/crio-63581869568f411b94f14232150c1191cd2732c90a21f9fe135991f5f0233b82 WatchSource:0}: Error finding container 63581869568f411b94f14232150c1191cd2732c90a21f9fe135991f5f0233b82: Status 404 returned error can't find the container with id 63581869568f411b94f14232150c1191cd2732c90a21f9fe135991f5f0233b82 Sep 05 08:06:23.630039 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:23.630004 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" event={"ID":"bf0cb253-b79a-4906-8986-fcbdd291d8f9","Type":"ContainerStarted","Data":"55bc924ca617065291fb1328e7722ca121be543b87c9e8a1f9835b753e2938a0"} Sep 05 08:06:23.630240 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:23.630068 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" event={"ID":"bf0cb253-b79a-4906-8986-fcbdd291d8f9","Type":"ContainerStarted","Data":"63581869568f411b94f14232150c1191cd2732c90a21f9fe135991f5f0233b82"} Sep 05 08:06:23.630240 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:23.630099 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" Sep 05 08:06:23.632027 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:23.631998 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" event={"ID":"8872c321-9589-4d4e-b78f-1dcfa38b9462","Type":"ContainerStarted","Data":"17e52ee8a5f4f632d954a14959f581960d4c272fb182d5eee654b36e05deb565"} Sep 05 08:06:23.632166 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:23.632050 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" event={"ID":"8872c321-9589-4d4e-b78f-1dcfa38b9462","Type":"ContainerStarted","Data":"2f99e4221a8031b3435ade933eb6995899604dc4d1508cc40447125b72763818"} Sep 05 08:06:23.632166 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:23.632080 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" Sep 05 08:06:23.633622 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:23.633598 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" event={"ID":"ee57f33b-65e7-459d-93b9-76292560fe93","Type":"ContainerStarted","Data":"65b3b56fae8dc8c366de22d51a6e6f223cb1138379286f1fc58ff5ee48b9e0bd"} Sep 05 08:06:23.633740 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:23.633639 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" event={"ID":"ee57f33b-65e7-459d-93b9-76292560fe93","Type":"ContainerStarted","Data":"a8a3009ec743f82ab6238863827085b5ba1fd1f1c3294cbf6017a06a4f12e78e"} Sep 05 08:06:23.633740 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:23.633670 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:23.648522 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:23.648440 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" podStartSLOduration=1.6484202959999998 podStartE2EDuration="1.648420296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:06:23.645861754 +0000 UTC m=+1086.254836876" watchObservedRunningTime="2026-09-05 08:06:23.648420296 +0000 UTC m=+1086.257395433" Sep 05 08:06:23.665642 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:23.665597 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" podStartSLOduration=2.318085019 podStartE2EDuration="2.665582512s" totalImagesPullingTime="347.497493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:06:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:06:23.66339011 +0000 UTC m=+1086.272365245" watchObservedRunningTime="2026-09-05 08:06:23.665582512 +0000 UTC m=+1086.274557689" Sep 05 08:06:23.682098 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:23.682052 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" podStartSLOduration=1.6820379540000001 podStartE2EDuration="1.682037954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:06:23.679935177 +0000 UTC m=+1086.288910299" watchObservedRunningTime="2026-09-05 08:06:23.682037954 +0000 UTC m=+1086.291013077" Sep 05 08:06:24.575160 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:24.575131 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" Sep 05 08:06:24.575612 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:24.575200 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" Sep 05 08:06:29.645228 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:29.645177 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:06:30.055185 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:30.055095 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:06:34.642068 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:34.642027 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" Sep 05 08:06:34.642537 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:34.642118 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" Sep 05 08:06:41.779792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.779756 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r"] Sep 05 08:06:41.784550 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.784524 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" Sep 05 08:06:41.786789 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.786766 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-cec446-dockercfg-l48bd\"" Sep 05 08:06:41.786919 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.786814 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-cec446\"" Sep 05 08:06:41.795083 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.795050 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r"] Sep 05 08:06:41.865418 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.865384 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn"] Sep 05 08:06:41.868961 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.868944 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" Sep 05 08:06:41.875723 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.875692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b546999-3cc4-4cc4-9ab2-d70470494ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r\" (UID: \"8b546999-3cc4-4cc4-9ab2-d70470494ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" Sep 05 08:06:41.875858 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.875780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxmtj\" (UniqueName: \"kubernetes.io/projected/8b546999-3cc4-4cc4-9ab2-d70470494ba8-kube-api-access-jxmtj\") pod \"payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r\" (UID: \"8b546999-3cc4-4cc4-9ab2-d70470494ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" Sep 05 08:06:41.878813 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.878791 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn"] Sep 05 08:06:41.976477 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.976425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn7sv\" (UniqueName: \"kubernetes.io/projected/3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35-kube-api-access-rn7sv\") pod \"payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn\" (UID: \"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" Sep 05 08:06:41.976477 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.976492 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b546999-3cc4-4cc4-9ab2-d70470494ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r\" (UID: \"8b546999-3cc4-4cc4-9ab2-d70470494ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" Sep 05 08:06:41.976758 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.976537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn\" (UID: \"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" Sep 05 08:06:41.976758 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.976572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxmtj\" (UniqueName: \"kubernetes.io/projected/8b546999-3cc4-4cc4-9ab2-d70470494ba8-kube-api-access-jxmtj\") pod \"payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r\" (UID: \"8b546999-3cc4-4cc4-9ab2-d70470494ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" Sep 05 08:06:41.977321 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.977296 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b546999-3cc4-4cc4-9ab2-d70470494ba8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r\" (UID: \"8b546999-3cc4-4cc4-9ab2-d70470494ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" Sep 05 08:06:41.988508 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:41.988480 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxmtj\" (UniqueName: \"kubernetes.io/projected/8b546999-3cc4-4cc4-9ab2-d70470494ba8-kube-api-access-jxmtj\") pod \"payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r\" (UID: \"8b546999-3cc4-4cc4-9ab2-d70470494ba8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" Sep 05 08:06:42.078027 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.077994 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn7sv\" (UniqueName: \"kubernetes.io/projected/3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35-kube-api-access-rn7sv\") pod \"payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn\" (UID: \"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" Sep 05 08:06:42.078195 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.078056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn\" (UID: \"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" Sep 05 08:06:42.078730 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.078712 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn\" (UID: \"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" Sep 05 08:06:42.086364 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.086336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn7sv\" (UniqueName: \"kubernetes.io/projected/3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35-kube-api-access-rn7sv\") pod \"payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn\" (UID: \"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" Sep 05 08:06:42.097362 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.097334 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" Sep 05 08:06:42.179585 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.179550 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" Sep 05 08:06:42.374763 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.374736 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r"] Sep 05 08:06:42.376118 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:06:42.376088 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b546999_3cc4_4cc4_9ab2_d70470494ba8.slice/crio-4b1cff6dc765c5e12e5369484f8d9d76158de2a4285e07a5584c4f220d49b98b WatchSource:0}: Error finding container 4b1cff6dc765c5e12e5369484f8d9d76158de2a4285e07a5584c4f220d49b98b: Status 404 returned error can't find the container with id 4b1cff6dc765c5e12e5369484f8d9d76158de2a4285e07a5584c4f220d49b98b Sep 05 08:06:42.459324 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.459294 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn"] Sep 05 08:06:42.460293 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:06:42.460256 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ccfe075_4ca3_4ca3_a2a1_dcf5d0837c35.slice/crio-f2e08d2c2b53e3bd61aa6fed374ee0e3e8d94902a9896b8b128c7fa489b923bb WatchSource:0}: Error finding container f2e08d2c2b53e3bd61aa6fed374ee0e3e8d94902a9896b8b128c7fa489b923bb: Status 404 returned error can't find the container with id f2e08d2c2b53e3bd61aa6fed374ee0e3e8d94902a9896b8b128c7fa489b923bb Sep 05 08:06:42.753667 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.753566 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" event={"ID":"8b546999-3cc4-4cc4-9ab2-d70470494ba8","Type":"ContainerStarted","Data":"37de408eca57e7f3c146c932818779b8623af7f4f82576dae9af21114464e257"} Sep 05 08:06:42.753667 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.753630 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" event={"ID":"8b546999-3cc4-4cc4-9ab2-d70470494ba8","Type":"ContainerStarted","Data":"4b1cff6dc765c5e12e5369484f8d9d76158de2a4285e07a5584c4f220d49b98b"} Sep 05 08:06:42.753667 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.753660 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" Sep 05 08:06:42.758433 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.758403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" event={"ID":"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35","Type":"ContainerStarted","Data":"57ea13c084c9f5a2012caf0e5007777f7e700133b76b3c09f541eb7007c39fd4"} Sep 05 08:06:42.758599 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.758453 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" event={"ID":"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35","Type":"ContainerStarted","Data":"f2e08d2c2b53e3bd61aa6fed374ee0e3e8d94902a9896b8b128c7fa489b923bb"} Sep 05 08:06:42.758599 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.758566 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" Sep 05 08:06:42.773654 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.773604 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" podStartSLOduration=1.773585215 podStartE2EDuration="1.773585215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:06:42.771190997 +0000 UTC m=+1105.380166146" watchObservedRunningTime="2026-09-05 08:06:42.773585215 +0000 UTC m=+1105.382560341" Sep 05 08:06:42.789349 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:42.789297 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" podStartSLOduration=1.7892811709999998 podStartE2EDuration="1.789281171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:06:42.787689145 +0000 UTC m=+1105.396664273" watchObservedRunningTime="2026-09-05 08:06:42.789281171 +0000 UTC m=+1105.398256293" Sep 05 08:06:45.851110 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:45.851070 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:06:50.972836 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:50.972800 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp"] Sep 05 08:06:50.976760 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:50.976743 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" Sep 05 08:06:50.979039 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:50.979018 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5ef325\"" Sep 05 08:06:50.979163 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:50.979049 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-5ef325-dockercfg-dm67w\"" Sep 05 08:06:50.984671 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:50.984646 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp"] Sep 05 08:06:51.057339 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.057306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f7ea83-8b96-4c21-9b1b-4588968fcf6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp\" (UID: \"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" Sep 05 08:06:51.057339 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.057351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxc52\" (UniqueName: \"kubernetes.io/projected/f4f7ea83-8b96-4c21-9b1b-4588968fcf6b-kube-api-access-gxc52\") pod \"payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp\" (UID: \"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" Sep 05 08:06:51.074132 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.074099 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf"] Sep 05 08:06:51.077681 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.077664 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" Sep 05 08:06:51.088304 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.088273 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf"] Sep 05 08:06:51.158286 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.158243 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f7ea83-8b96-4c21-9b1b-4588968fcf6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp\" (UID: \"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" Sep 05 08:06:51.158512 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.158300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxc52\" (UniqueName: \"kubernetes.io/projected/f4f7ea83-8b96-4c21-9b1b-4588968fcf6b-kube-api-access-gxc52\") pod \"payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp\" (UID: \"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" Sep 05 08:06:51.158512 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.158339 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvkpk\" (UniqueName: \"kubernetes.io/projected/78460e86-f41b-4f89-9d46-591534d4b527-kube-api-access-jvkpk\") pod \"payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf\" (UID: \"78460e86-f41b-4f89-9d46-591534d4b527\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" Sep 05 08:06:51.158512 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.158399 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78460e86-f41b-4f89-9d46-591534d4b527-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf\" (UID: \"78460e86-f41b-4f89-9d46-591534d4b527\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" Sep 05 08:06:51.158971 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.158949 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f7ea83-8b96-4c21-9b1b-4588968fcf6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp\" (UID: \"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" Sep 05 08:06:51.168253 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.168196 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxc52\" (UniqueName: \"kubernetes.io/projected/f4f7ea83-8b96-4c21-9b1b-4588968fcf6b-kube-api-access-gxc52\") pod \"payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp\" (UID: \"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" Sep 05 08:06:51.259555 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.259424 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvkpk\" (UniqueName: \"kubernetes.io/projected/78460e86-f41b-4f89-9d46-591534d4b527-kube-api-access-jvkpk\") pod \"payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf\" (UID: \"78460e86-f41b-4f89-9d46-591534d4b527\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" Sep 05 08:06:51.259555 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.259547 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78460e86-f41b-4f89-9d46-591534d4b527-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf\" (UID: \"78460e86-f41b-4f89-9d46-591534d4b527\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" Sep 05 08:06:51.260355 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.260336 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78460e86-f41b-4f89-9d46-591534d4b527-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf\" (UID: \"78460e86-f41b-4f89-9d46-591534d4b527\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" Sep 05 08:06:51.268015 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.267994 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvkpk\" (UniqueName: \"kubernetes.io/projected/78460e86-f41b-4f89-9d46-591534d4b527-kube-api-access-jvkpk\") pod \"payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf\" (UID: \"78460e86-f41b-4f89-9d46-591534d4b527\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" Sep 05 08:06:51.290025 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.289992 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" Sep 05 08:06:51.389313 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.389279 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" Sep 05 08:06:51.567875 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.567851 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp"] Sep 05 08:06:51.569326 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:06:51.569296 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4f7ea83_8b96_4c21_9b1b_4588968fcf6b.slice/crio-1d119824926c34a50b114763ff675c04537a0dd167f30b6cc94148114e0faa6f WatchSource:0}: Error finding container 1d119824926c34a50b114763ff675c04537a0dd167f30b6cc94148114e0faa6f: Status 404 returned error can't find the container with id 1d119824926c34a50b114763ff675c04537a0dd167f30b6cc94148114e0faa6f Sep 05 08:06:51.670151 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.670127 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf"] Sep 05 08:06:51.672361 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:06:51.672332 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78460e86_f41b_4f89_9d46_591534d4b527.slice/crio-8857cca39a0e9a64e25bc8fc00d5db4c0fde19f919a363669f41efe5b2e31928 WatchSource:0}: Error finding container 8857cca39a0e9a64e25bc8fc00d5db4c0fde19f919a363669f41efe5b2e31928: Status 404 returned error can't find the container with id 8857cca39a0e9a64e25bc8fc00d5db4c0fde19f919a363669f41efe5b2e31928 Sep 05 08:06:51.816403 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.816303 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" event={"ID":"78460e86-f41b-4f89-9d46-591534d4b527","Type":"ContainerStarted","Data":"f02b97e9b3e3330798a43a89bcfe7e26c287cf56c41548ac0a94367e9e45e29e"} Sep 05 08:06:51.816403 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.816373 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" event={"ID":"78460e86-f41b-4f89-9d46-591534d4b527","Type":"ContainerStarted","Data":"8857cca39a0e9a64e25bc8fc00d5db4c0fde19f919a363669f41efe5b2e31928"} Sep 05 08:06:51.816403 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.816403 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" Sep 05 08:06:51.818639 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.818607 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" event={"ID":"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b","Type":"ContainerStarted","Data":"d7a46475fcdd27803e58f3591416061f69424903add993504b0782b4dfb9e698"} Sep 05 08:06:51.818800 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.818667 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" Sep 05 08:06:51.818800 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.818690 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" event={"ID":"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b","Type":"ContainerStarted","Data":"1d119824926c34a50b114763ff675c04537a0dd167f30b6cc94148114e0faa6f"} Sep 05 08:06:51.833301 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.833244 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" podStartSLOduration=0.83319616 podStartE2EDuration="833.19616ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:06:51.831331368 +0000 UTC m=+1114.440306492" watchObservedRunningTime="2026-09-05 08:06:51.83319616 +0000 UTC m=+1114.442171287" Sep 05 08:06:51.849581 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:51.849530 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" podStartSLOduration=1.849515363 podStartE2EDuration="1.849515363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:06:51.846909467 +0000 UTC m=+1114.455884589" watchObservedRunningTime="2026-09-05 08:06:51.849515363 +0000 UTC m=+1114.458490536" Sep 05 08:06:53.769708 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:53.769678 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" Sep 05 08:06:53.770127 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:53.769759 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" Sep 05 08:06:55.453739 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:06:55.453701 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:07:02.828328 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:02.828294 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" Sep 05 08:07:02.828750 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:02.828370 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" Sep 05 08:07:23.007220 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.007174 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n"] Sep 05 08:07:23.011774 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.011756 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" Sep 05 08:07:23.014852 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.014820 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4b62745d\"" Sep 05 08:07:23.015015 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.014998 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4b62745d-dockercfg-26s5t\"" Sep 05 08:07:23.020838 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.020818 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n"] Sep 05 08:07:23.127788 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.127752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-764pn\" (UniqueName: \"kubernetes.io/projected/61718d71-b9c2-4235-9008-bc118c53cd3f-kube-api-access-764pn\") pod \"payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n\" (UID: \"61718d71-b9c2-4235-9008-bc118c53cd3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" Sep 05 08:07:23.127958 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.127886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61718d71-b9c2-4235-9008-bc118c53cd3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n\" (UID: \"61718d71-b9c2-4235-9008-bc118c53cd3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" Sep 05 08:07:23.228643 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.228615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61718d71-b9c2-4235-9008-bc118c53cd3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n\" (UID: \"61718d71-b9c2-4235-9008-bc118c53cd3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" Sep 05 08:07:23.228833 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.228661 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-764pn\" (UniqueName: \"kubernetes.io/projected/61718d71-b9c2-4235-9008-bc118c53cd3f-kube-api-access-764pn\") pod \"payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n\" (UID: \"61718d71-b9c2-4235-9008-bc118c53cd3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" Sep 05 08:07:23.229338 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.229317 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61718d71-b9c2-4235-9008-bc118c53cd3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n\" (UID: \"61718d71-b9c2-4235-9008-bc118c53cd3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" Sep 05 08:07:23.240352 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.240323 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-764pn\" (UniqueName: \"kubernetes.io/projected/61718d71-b9c2-4235-9008-bc118c53cd3f-kube-api-access-764pn\") pod \"payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n\" (UID: \"61718d71-b9c2-4235-9008-bc118c53cd3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" Sep 05 08:07:23.323036 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.322995 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" Sep 05 08:07:23.597431 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.597407 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n"] Sep 05 08:07:23.599657 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:07:23.599635 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61718d71_b9c2_4235_9008_bc118c53cd3f.slice/crio-140a85707c6d633d6941c6746b0be635b05e97baaeda29467d125081258d22fe WatchSource:0}: Error finding container 140a85707c6d633d6941c6746b0be635b05e97baaeda29467d125081258d22fe: Status 404 returned error can't find the container with id 140a85707c6d633d6941c6746b0be635b05e97baaeda29467d125081258d22fe Sep 05 08:07:23.998127 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.998025 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" event={"ID":"61718d71-b9c2-4235-9008-bc118c53cd3f","Type":"ContainerStarted","Data":"d99bc1c40dcdc4ea9043f1f4b4348c1cfb83d320dbba7adb3883350f9992da52"} Sep 05 08:07:23.998127 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.998102 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" event={"ID":"61718d71-b9c2-4235-9008-bc118c53cd3f","Type":"ContainerStarted","Data":"140a85707c6d633d6941c6746b0be635b05e97baaeda29467d125081258d22fe"} Sep 05 08:07:23.998362 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:23.998229 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" Sep 05 08:07:24.016388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:24.016341 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" podStartSLOduration=2.016327646 podStartE2EDuration="2.016327646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:07:24.014304223 +0000 UTC m=+1146.623279345" watchObservedRunningTime="2026-09-05 08:07:24.016327646 +0000 UTC m=+1146.625302834" Sep 05 08:07:28.658985 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:28.658949 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:07:35.005301 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:35.005271 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" Sep 05 08:07:40.048402 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:07:40.048313 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:08:04.855363 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:04.855321 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:08:17.940649 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:17.940620 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 08:08:17.942161 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:17.942137 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 08:08:17.944193 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:17.944170 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 08:08:17.945414 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:17.945395 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 08:08:19.054510 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:19.054471 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:08:33.715825 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:33.715788 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:08:46.912128 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:46.912064 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n"] Sep 05 08:08:46.913004 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:46.912942 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" podUID="61718d71-b9c2-4235-9008-bc118c53cd3f" containerName="payload-pre-processing" containerID="cri-o://d99bc1c40dcdc4ea9043f1f4b4348c1cfb83d320dbba7adb3883350f9992da52" gracePeriod=30 Sep 05 08:08:47.180059 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.179976 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" Sep 05 08:08:47.275236 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.275183 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-764pn\" (UniqueName: \"kubernetes.io/projected/61718d71-b9c2-4235-9008-bc118c53cd3f-kube-api-access-764pn\") pod \"61718d71-b9c2-4235-9008-bc118c53cd3f\" (UID: \"61718d71-b9c2-4235-9008-bc118c53cd3f\") " Sep 05 08:08:47.275397 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.275247 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61718d71-b9c2-4235-9008-bc118c53cd3f-plugins-config-volume\") pod \"61718d71-b9c2-4235-9008-bc118c53cd3f\" (UID: \"61718d71-b9c2-4235-9008-bc118c53cd3f\") " Sep 05 08:08:47.275655 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.275631 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61718d71-b9c2-4235-9008-bc118c53cd3f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "61718d71-b9c2-4235-9008-bc118c53cd3f" (UID: "61718d71-b9c2-4235-9008-bc118c53cd3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:08:47.277597 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.277566 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61718d71-b9c2-4235-9008-bc118c53cd3f-kube-api-access-764pn" (OuterVolumeSpecName: "kube-api-access-764pn") pod "61718d71-b9c2-4235-9008-bc118c53cd3f" (UID: "61718d71-b9c2-4235-9008-bc118c53cd3f"). InnerVolumeSpecName "kube-api-access-764pn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:08:47.376903 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.376863 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-764pn\" (UniqueName: \"kubernetes.io/projected/61718d71-b9c2-4235-9008-bc118c53cd3f-kube-api-access-764pn\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:08:47.376903 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.376894 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61718d71-b9c2-4235-9008-bc118c53cd3f-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:08:47.474030 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.473935 2576 generic.go:356] "Generic (PLEG): container finished" podID="61718d71-b9c2-4235-9008-bc118c53cd3f" containerID="d99bc1c40dcdc4ea9043f1f4b4348c1cfb83d320dbba7adb3883350f9992da52" exitCode=0 Sep 05 08:08:47.474030 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.473982 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" Sep 05 08:08:47.474030 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.474014 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" event={"ID":"61718d71-b9c2-4235-9008-bc118c53cd3f","Type":"ContainerDied","Data":"d99bc1c40dcdc4ea9043f1f4b4348c1cfb83d320dbba7adb3883350f9992da52"} Sep 05 08:08:47.474281 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.474070 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n" event={"ID":"61718d71-b9c2-4235-9008-bc118c53cd3f","Type":"ContainerDied","Data":"140a85707c6d633d6941c6746b0be635b05e97baaeda29467d125081258d22fe"} Sep 05 08:08:47.474281 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.474094 2576 scope.go:117] "RemoveContainer" containerID="d99bc1c40dcdc4ea9043f1f4b4348c1cfb83d320dbba7adb3883350f9992da52" Sep 05 08:08:47.486727 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.486702 2576 scope.go:117] "RemoveContainer" containerID="d99bc1c40dcdc4ea9043f1f4b4348c1cfb83d320dbba7adb3883350f9992da52" Sep 05 08:08:47.487093 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:08:47.487067 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d99bc1c40dcdc4ea9043f1f4b4348c1cfb83d320dbba7adb3883350f9992da52\": container with ID starting with d99bc1c40dcdc4ea9043f1f4b4348c1cfb83d320dbba7adb3883350f9992da52 not found: ID does not exist" containerID="d99bc1c40dcdc4ea9043f1f4b4348c1cfb83d320dbba7adb3883350f9992da52" Sep 05 08:08:47.487240 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.487106 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d99bc1c40dcdc4ea9043f1f4b4348c1cfb83d320dbba7adb3883350f9992da52"} err="failed to get container status \"d99bc1c40dcdc4ea9043f1f4b4348c1cfb83d320dbba7adb3883350f9992da52\": rpc error: code = NotFound desc = could not find container \"d99bc1c40dcdc4ea9043f1f4b4348c1cfb83d320dbba7adb3883350f9992da52\": container with ID starting with d99bc1c40dcdc4ea9043f1f4b4348c1cfb83d320dbba7adb3883350f9992da52 not found: ID does not exist" Sep 05 08:08:47.497590 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.497563 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n"] Sep 05 08:08:47.502748 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.502722 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4b62745d-7f49c69494-l4x5n"] Sep 05 08:08:47.887555 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:47.887512 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61718d71-b9c2-4235-9008-bc118c53cd3f" path="/var/lib/kubelet/pods/61718d71-b9c2-4235-9008-bc118c53cd3f/volumes" Sep 05 08:08:52.162159 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:08:52.162121 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:09:09.346387 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:09:09.346299 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:09:24.048347 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:09:24.048309 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:09:40.846698 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:09:40.846662 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:09:58.150119 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:09:58.150082 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:10:15.544821 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:15.544781 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:10:24.854026 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:24.853990 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:10:52.051841 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.051752 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk"] Sep 05 08:10:52.052353 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.052156 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61718d71-b9c2-4235-9008-bc118c53cd3f" containerName="payload-pre-processing" Sep 05 08:10:52.052353 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.052167 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="61718d71-b9c2-4235-9008-bc118c53cd3f" containerName="payload-pre-processing" Sep 05 08:10:52.052353 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.052260 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="61718d71-b9c2-4235-9008-bc118c53cd3f" containerName="payload-pre-processing" Sep 05 08:10:52.055179 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.055162 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" Sep 05 08:10:52.057486 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.057465 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e171e4-dockercfg-82kmm\"" Sep 05 08:10:52.057644 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.057497 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e171e4\"" Sep 05 08:10:52.063590 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.063564 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk"] Sep 05 08:10:52.214960 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.214910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2375855-4588-4d7e-b3e6-bfb61a9f9342-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk\" (UID: \"a2375855-4588-4d7e-b3e6-bfb61a9f9342\") " pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" Sep 05 08:10:52.215165 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.214989 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz2tg\" (UniqueName: \"kubernetes.io/projected/a2375855-4588-4d7e-b3e6-bfb61a9f9342-kube-api-access-hz2tg\") pod \"payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk\" (UID: \"a2375855-4588-4d7e-b3e6-bfb61a9f9342\") " pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" Sep 05 08:10:52.315945 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.315905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2375855-4588-4d7e-b3e6-bfb61a9f9342-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk\" (UID: \"a2375855-4588-4d7e-b3e6-bfb61a9f9342\") " pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" Sep 05 08:10:52.316115 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.315985 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz2tg\" (UniqueName: \"kubernetes.io/projected/a2375855-4588-4d7e-b3e6-bfb61a9f9342-kube-api-access-hz2tg\") pod \"payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk\" (UID: \"a2375855-4588-4d7e-b3e6-bfb61a9f9342\") " pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" Sep 05 08:10:52.316659 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.316638 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2375855-4588-4d7e-b3e6-bfb61a9f9342-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk\" (UID: \"a2375855-4588-4d7e-b3e6-bfb61a9f9342\") " pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" Sep 05 08:10:52.323736 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.323709 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz2tg\" (UniqueName: \"kubernetes.io/projected/a2375855-4588-4d7e-b3e6-bfb61a9f9342-kube-api-access-hz2tg\") pod \"payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk\" (UID: \"a2375855-4588-4d7e-b3e6-bfb61a9f9342\") " pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" Sep 05 08:10:52.369490 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.369453 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" Sep 05 08:10:52.648187 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:52.648164 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk"] Sep 05 08:10:52.649842 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:10:52.649815 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2375855_4588_4d7e_b3e6_bfb61a9f9342.slice/crio-5dfd10e4877f38f25bebf4f1322faebf300f63f37ced91f38e4c5918aa2fa67d WatchSource:0}: Error finding container 5dfd10e4877f38f25bebf4f1322faebf300f63f37ced91f38e4c5918aa2fa67d: Status 404 returned error can't find the container with id 5dfd10e4877f38f25bebf4f1322faebf300f63f37ced91f38e4c5918aa2fa67d Sep 05 08:10:53.179305 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:53.179270 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" event={"ID":"a2375855-4588-4d7e-b3e6-bfb61a9f9342","Type":"ContainerStarted","Data":"ec2c9c47b3392de5c7b913a13d035c89d3a241c1a475a1f35f8792f6d89fa837"} Sep 05 08:10:53.179305 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:53.179324 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" event={"ID":"a2375855-4588-4d7e-b3e6-bfb61a9f9342","Type":"ContainerStarted","Data":"5dfd10e4877f38f25bebf4f1322faebf300f63f37ced91f38e4c5918aa2fa67d"} Sep 05 08:10:53.179770 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:53.179370 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" Sep 05 08:10:53.197547 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:53.197496 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" podStartSLOduration=1.1974814870000001 podStartE2EDuration="1.197481487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:10:53.195515454 +0000 UTC m=+1355.804490578" watchObservedRunningTime="2026-09-05 08:10:53.197481487 +0000 UTC m=+1355.806456610" Sep 05 08:10:58.754041 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:10:58.754005 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:11:04.187627 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:04.187594 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" Sep 05 08:11:11.244765 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:11.244733 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:11:21.237544 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:21.237498 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29"] Sep 05 08:11:21.240954 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:21.240930 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" Sep 05 08:11:21.242962 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:21.242939 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d850e3\"" Sep 05 08:11:21.243083 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:21.242939 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d850e3-dockercfg-wjcbc\"" Sep 05 08:11:21.250437 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:21.250410 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29"] Sep 05 08:11:21.368124 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:21.368092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xdlr\" (UniqueName: \"kubernetes.io/projected/4a9e11bd-bd86-45ee-a638-d5b834b25955-kube-api-access-4xdlr\") pod \"payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29\" (UID: \"4a9e11bd-bd86-45ee-a638-d5b834b25955\") " pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" Sep 05 08:11:21.368346 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:21.368178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9e11bd-bd86-45ee-a638-d5b834b25955-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29\" (UID: \"4a9e11bd-bd86-45ee-a638-d5b834b25955\") " pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" Sep 05 08:11:21.469069 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:21.469031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xdlr\" (UniqueName: \"kubernetes.io/projected/4a9e11bd-bd86-45ee-a638-d5b834b25955-kube-api-access-4xdlr\") pod \"payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29\" (UID: \"4a9e11bd-bd86-45ee-a638-d5b834b25955\") " pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" Sep 05 08:11:21.469274 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:21.469115 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9e11bd-bd86-45ee-a638-d5b834b25955-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29\" (UID: \"4a9e11bd-bd86-45ee-a638-d5b834b25955\") " pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" Sep 05 08:11:21.469823 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:21.469803 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9e11bd-bd86-45ee-a638-d5b834b25955-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29\" (UID: \"4a9e11bd-bd86-45ee-a638-d5b834b25955\") " pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" Sep 05 08:11:21.477275 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:21.477246 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xdlr\" (UniqueName: \"kubernetes.io/projected/4a9e11bd-bd86-45ee-a638-d5b834b25955-kube-api-access-4xdlr\") pod \"payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29\" (UID: \"4a9e11bd-bd86-45ee-a638-d5b834b25955\") " pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" Sep 05 08:11:21.552880 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:21.552789 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" Sep 05 08:11:21.827521 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:21.827479 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29"] Sep 05 08:11:21.829488 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:11:21.829455 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9e11bd_bd86_45ee_a638_d5b834b25955.slice/crio-4a7fd3c4ad234ee14275ec8c9685821fd6d88f0e73a701cb429dd3c8105cd53a WatchSource:0}: Error finding container 4a7fd3c4ad234ee14275ec8c9685821fd6d88f0e73a701cb429dd3c8105cd53a: Status 404 returned error can't find the container with id 4a7fd3c4ad234ee14275ec8c9685821fd6d88f0e73a701cb429dd3c8105cd53a Sep 05 08:11:22.349012 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:22.348973 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" event={"ID":"4a9e11bd-bd86-45ee-a638-d5b834b25955","Type":"ContainerStarted","Data":"4c1af4dd980da48ad83ede9423c39b102a5e4523b29a18fed3f080191a63f1d9"} Sep 05 08:11:22.349545 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:22.349070 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" Sep 05 08:11:22.349545 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:22.349088 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" event={"ID":"4a9e11bd-bd86-45ee-a638-d5b834b25955","Type":"ContainerStarted","Data":"4a7fd3c4ad234ee14275ec8c9685821fd6d88f0e73a701cb429dd3c8105cd53a"} Sep 05 08:11:22.365954 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:22.365898 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" podStartSLOduration=1.365881489 podStartE2EDuration="1.365881489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:11:22.363228955 +0000 UTC m=+1384.972204077" watchObservedRunningTime="2026-09-05 08:11:22.365881489 +0000 UTC m=+1384.974856613" Sep 05 08:11:30.052008 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:30.051970 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:11:33.357958 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:33.357923 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" Sep 05 08:11:47.749139 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:11:47.749102 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:12:03.199646 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.199609 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk"] Sep 05 08:12:03.200117 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.199899 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" podUID="a2375855-4588-4d7e-b3e6-bfb61a9f9342" containerName="payload-processing" containerID="cri-o://ec2c9c47b3392de5c7b913a13d035c89d3a241c1a475a1f35f8792f6d89fa837" gracePeriod=30 Sep 05 08:12:03.470550 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.470521 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" Sep 05 08:12:03.547202 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.547166 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2375855-4588-4d7e-b3e6-bfb61a9f9342-plugins-config-volume\") pod \"a2375855-4588-4d7e-b3e6-bfb61a9f9342\" (UID: \"a2375855-4588-4d7e-b3e6-bfb61a9f9342\") " Sep 05 08:12:03.547398 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.547262 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz2tg\" (UniqueName: \"kubernetes.io/projected/a2375855-4588-4d7e-b3e6-bfb61a9f9342-kube-api-access-hz2tg\") pod \"a2375855-4588-4d7e-b3e6-bfb61a9f9342\" (UID: \"a2375855-4588-4d7e-b3e6-bfb61a9f9342\") " Sep 05 08:12:03.547584 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.547561 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2375855-4588-4d7e-b3e6-bfb61a9f9342-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2375855-4588-4d7e-b3e6-bfb61a9f9342" (UID: "a2375855-4588-4d7e-b3e6-bfb61a9f9342"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:12:03.549424 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.549403 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2375855-4588-4d7e-b3e6-bfb61a9f9342-kube-api-access-hz2tg" (OuterVolumeSpecName: "kube-api-access-hz2tg") pod "a2375855-4588-4d7e-b3e6-bfb61a9f9342" (UID: "a2375855-4588-4d7e-b3e6-bfb61a9f9342"). InnerVolumeSpecName "kube-api-access-hz2tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:12:03.585741 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.585712 2576 generic.go:356] "Generic (PLEG): container finished" podID="a2375855-4588-4d7e-b3e6-bfb61a9f9342" containerID="ec2c9c47b3392de5c7b913a13d035c89d3a241c1a475a1f35f8792f6d89fa837" exitCode=0 Sep 05 08:12:03.585896 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.585762 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" Sep 05 08:12:03.585896 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.585774 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" event={"ID":"a2375855-4588-4d7e-b3e6-bfb61a9f9342","Type":"ContainerDied","Data":"ec2c9c47b3392de5c7b913a13d035c89d3a241c1a475a1f35f8792f6d89fa837"} Sep 05 08:12:03.585896 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.585815 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk" event={"ID":"a2375855-4588-4d7e-b3e6-bfb61a9f9342","Type":"ContainerDied","Data":"5dfd10e4877f38f25bebf4f1322faebf300f63f37ced91f38e4c5918aa2fa67d"} Sep 05 08:12:03.585896 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.585837 2576 scope.go:117] "RemoveContainer" containerID="ec2c9c47b3392de5c7b913a13d035c89d3a241c1a475a1f35f8792f6d89fa837" Sep 05 08:12:03.595945 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.595925 2576 scope.go:117] "RemoveContainer" containerID="ec2c9c47b3392de5c7b913a13d035c89d3a241c1a475a1f35f8792f6d89fa837" Sep 05 08:12:03.596201 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:12:03.596185 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2c9c47b3392de5c7b913a13d035c89d3a241c1a475a1f35f8792f6d89fa837\": container with ID starting with ec2c9c47b3392de5c7b913a13d035c89d3a241c1a475a1f35f8792f6d89fa837 not found: ID does not exist" containerID="ec2c9c47b3392de5c7b913a13d035c89d3a241c1a475a1f35f8792f6d89fa837" Sep 05 08:12:03.596355 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.596227 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2c9c47b3392de5c7b913a13d035c89d3a241c1a475a1f35f8792f6d89fa837"} err="failed to get container status \"ec2c9c47b3392de5c7b913a13d035c89d3a241c1a475a1f35f8792f6d89fa837\": rpc error: code = NotFound desc = could not find container \"ec2c9c47b3392de5c7b913a13d035c89d3a241c1a475a1f35f8792f6d89fa837\": container with ID starting with ec2c9c47b3392de5c7b913a13d035c89d3a241c1a475a1f35f8792f6d89fa837 not found: ID does not exist" Sep 05 08:12:03.607616 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.607590 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk"] Sep 05 08:12:03.611146 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.611127 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e171e4-68c7fd9bf-2nllk"] Sep 05 08:12:03.648063 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.648036 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2375855-4588-4d7e-b3e6-bfb61a9f9342-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:12:03.648063 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.648059 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz2tg\" (UniqueName: \"kubernetes.io/projected/a2375855-4588-4d7e-b3e6-bfb61a9f9342-kube-api-access-hz2tg\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:12:03.887574 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:03.887542 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2375855-4588-4d7e-b3e6-bfb61a9f9342" path="/var/lib/kubelet/pods/a2375855-4588-4d7e-b3e6-bfb61a9f9342/volumes" Sep 05 08:12:07.551791 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:07.551757 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:12:24.338302 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:24.338262 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:12:26.027703 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.027653 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29"] Sep 05 08:12:26.028228 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.027941 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" podUID="4a9e11bd-bd86-45ee-a638-d5b834b25955" containerName="payload-processing" containerID="cri-o://4c1af4dd980da48ad83ede9423c39b102a5e4523b29a18fed3f080191a63f1d9" gracePeriod=30 Sep 05 08:12:26.296030 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.296007 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" Sep 05 08:12:26.443667 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.443635 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xdlr\" (UniqueName: \"kubernetes.io/projected/4a9e11bd-bd86-45ee-a638-d5b834b25955-kube-api-access-4xdlr\") pod \"4a9e11bd-bd86-45ee-a638-d5b834b25955\" (UID: \"4a9e11bd-bd86-45ee-a638-d5b834b25955\") " Sep 05 08:12:26.443848 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.443683 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9e11bd-bd86-45ee-a638-d5b834b25955-plugins-config-volume\") pod \"4a9e11bd-bd86-45ee-a638-d5b834b25955\" (UID: \"4a9e11bd-bd86-45ee-a638-d5b834b25955\") " Sep 05 08:12:26.444079 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.444056 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9e11bd-bd86-45ee-a638-d5b834b25955-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4a9e11bd-bd86-45ee-a638-d5b834b25955" (UID: "4a9e11bd-bd86-45ee-a638-d5b834b25955"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:12:26.445858 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.445840 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9e11bd-bd86-45ee-a638-d5b834b25955-kube-api-access-4xdlr" (OuterVolumeSpecName: "kube-api-access-4xdlr") pod "4a9e11bd-bd86-45ee-a638-d5b834b25955" (UID: "4a9e11bd-bd86-45ee-a638-d5b834b25955"). InnerVolumeSpecName "kube-api-access-4xdlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:12:26.545245 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.545130 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xdlr\" (UniqueName: \"kubernetes.io/projected/4a9e11bd-bd86-45ee-a638-d5b834b25955-kube-api-access-4xdlr\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:12:26.545245 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.545170 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a9e11bd-bd86-45ee-a638-d5b834b25955-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:12:26.718334 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.718298 2576 generic.go:356] "Generic (PLEG): container finished" podID="4a9e11bd-bd86-45ee-a638-d5b834b25955" containerID="4c1af4dd980da48ad83ede9423c39b102a5e4523b29a18fed3f080191a63f1d9" exitCode=0 Sep 05 08:12:26.718539 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.718342 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" Sep 05 08:12:26.718539 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.718388 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" event={"ID":"4a9e11bd-bd86-45ee-a638-d5b834b25955","Type":"ContainerDied","Data":"4c1af4dd980da48ad83ede9423c39b102a5e4523b29a18fed3f080191a63f1d9"} Sep 05 08:12:26.718539 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.718447 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29" event={"ID":"4a9e11bd-bd86-45ee-a638-d5b834b25955","Type":"ContainerDied","Data":"4a7fd3c4ad234ee14275ec8c9685821fd6d88f0e73a701cb429dd3c8105cd53a"} Sep 05 08:12:26.718539 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.718478 2576 scope.go:117] "RemoveContainer" containerID="4c1af4dd980da48ad83ede9423c39b102a5e4523b29a18fed3f080191a63f1d9" Sep 05 08:12:26.728532 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.728518 2576 scope.go:117] "RemoveContainer" containerID="4c1af4dd980da48ad83ede9423c39b102a5e4523b29a18fed3f080191a63f1d9" Sep 05 08:12:26.728843 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:12:26.728821 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c1af4dd980da48ad83ede9423c39b102a5e4523b29a18fed3f080191a63f1d9\": container with ID starting with 4c1af4dd980da48ad83ede9423c39b102a5e4523b29a18fed3f080191a63f1d9 not found: ID does not exist" containerID="4c1af4dd980da48ad83ede9423c39b102a5e4523b29a18fed3f080191a63f1d9" Sep 05 08:12:26.728938 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.728850 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c1af4dd980da48ad83ede9423c39b102a5e4523b29a18fed3f080191a63f1d9"} err="failed to get container status \"4c1af4dd980da48ad83ede9423c39b102a5e4523b29a18fed3f080191a63f1d9\": rpc error: code = NotFound desc = could not find container \"4c1af4dd980da48ad83ede9423c39b102a5e4523b29a18fed3f080191a63f1d9\": container with ID starting with 4c1af4dd980da48ad83ede9423c39b102a5e4523b29a18fed3f080191a63f1d9 not found: ID does not exist" Sep 05 08:12:26.738951 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.738927 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29"] Sep 05 08:12:26.744482 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:26.744451 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d850e3-64c6f78799-rkl29"] Sep 05 08:12:27.886872 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:27.886842 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a9e11bd-bd86-45ee-a638-d5b834b25955" path="/var/lib/kubelet/pods/4a9e11bd-bd86-45ee-a638-d5b834b25955/volumes" Sep 05 08:12:38.787579 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:38.787534 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:12:43.820023 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:43.819980 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h"] Sep 05 08:12:43.820553 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:43.820270 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" podUID="63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb" containerName="maas-api" containerID="cri-o://a2521b2b779edac51c360cd2370de15c960df70ff1c4d98c28267a947cb34e8e" gracePeriod=30 Sep 05 08:12:43.876996 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:43.876960 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx"] Sep 05 08:12:43.877267 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:43.877201 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" podUID="d592f3e0-53bd-4928-a47a-f53d54c6600d" containerName="payload-processing" containerID="cri-o://68d4b1a3f33824d21bc2bb208015aa3b5fec5c7c89ce45a5f42ff6cae28673f8" gracePeriod=30 Sep 05 08:12:43.897897 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:43.890924 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml"] Sep 05 08:12:43.897897 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:43.891203 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" podUID="cb0ed63c-a287-4d29-999f-0d6c9c950064" containerName="payload-pre-processing" containerID="cri-o://e23ca6bce8562dcfb634b02b3e059d0cb87717068042e6a5c63be86b53a94921" gracePeriod=30 Sep 05 08:12:44.156942 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.156914 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:12:44.163997 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.163974 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" Sep 05 08:12:44.185345 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.185317 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" Sep 05 08:12:44.309164 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.309129 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb0ed63c-a287-4d29-999f-0d6c9c950064-plugins-config-volume\") pod \"cb0ed63c-a287-4d29-999f-0d6c9c950064\" (UID: \"cb0ed63c-a287-4d29-999f-0d6c9c950064\") " Sep 05 08:12:44.309376 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.309187 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-metrics-certs\") pod \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " Sep 05 08:12:44.309376 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.309257 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s7m8\" (UniqueName: \"kubernetes.io/projected/cb0ed63c-a287-4d29-999f-0d6c9c950064-kube-api-access-7s7m8\") pod \"cb0ed63c-a287-4d29-999f-0d6c9c950064\" (UID: \"cb0ed63c-a287-4d29-999f-0d6c9c950064\") " Sep 05 08:12:44.309376 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.309306 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-maas-api-tls\") pod \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " Sep 05 08:12:44.309376 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.309335 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjzg5\" (UniqueName: \"kubernetes.io/projected/d592f3e0-53bd-4928-a47a-f53d54c6600d-kube-api-access-zjzg5\") pod \"d592f3e0-53bd-4928-a47a-f53d54c6600d\" (UID: \"d592f3e0-53bd-4928-a47a-f53d54c6600d\") " Sep 05 08:12:44.309376 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.309367 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lkgx\" (UniqueName: \"kubernetes.io/projected/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-kube-api-access-5lkgx\") pod \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " Sep 05 08:12:44.309670 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.309396 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d592f3e0-53bd-4928-a47a-f53d54c6600d-plugins-config-volume\") pod \"d592f3e0-53bd-4928-a47a-f53d54c6600d\" (UID: \"d592f3e0-53bd-4928-a47a-f53d54c6600d\") " Sep 05 08:12:44.309670 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.309458 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-openshift-service-ca\") pod \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\" (UID: \"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb\") " Sep 05 08:12:44.309670 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.309608 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0ed63c-a287-4d29-999f-0d6c9c950064-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb0ed63c-a287-4d29-999f-0d6c9c950064" (UID: "cb0ed63c-a287-4d29-999f-0d6c9c950064"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:12:44.309832 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.309733 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb0ed63c-a287-4d29-999f-0d6c9c950064-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:12:44.309989 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.309965 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d592f3e0-53bd-4928-a47a-f53d54c6600d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d592f3e0-53bd-4928-a47a-f53d54c6600d" (UID: "d592f3e0-53bd-4928-a47a-f53d54c6600d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:12:44.310238 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.310185 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb" (UID: "63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:12:44.311705 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.311680 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d592f3e0-53bd-4928-a47a-f53d54c6600d-kube-api-access-zjzg5" (OuterVolumeSpecName: "kube-api-access-zjzg5") pod "d592f3e0-53bd-4928-a47a-f53d54c6600d" (UID: "d592f3e0-53bd-4928-a47a-f53d54c6600d"). InnerVolumeSpecName "kube-api-access-zjzg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:12:44.311793 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.311710 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb" (UID: "63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:12:44.311953 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.311922 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-kube-api-access-5lkgx" (OuterVolumeSpecName: "kube-api-access-5lkgx") pod "63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb" (UID: "63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb"). InnerVolumeSpecName "kube-api-access-5lkgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:12:44.312032 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.312001 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb" (UID: "63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:12:44.312032 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.312015 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0ed63c-a287-4d29-999f-0d6c9c950064-kube-api-access-7s7m8" (OuterVolumeSpecName: "kube-api-access-7s7m8") pod "cb0ed63c-a287-4d29-999f-0d6c9c950064" (UID: "cb0ed63c-a287-4d29-999f-0d6c9c950064"). InnerVolumeSpecName "kube-api-access-7s7m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:12:44.411121 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.411036 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s7m8\" (UniqueName: \"kubernetes.io/projected/cb0ed63c-a287-4d29-999f-0d6c9c950064-kube-api-access-7s7m8\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:12:44.411121 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.411065 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-maas-api-tls\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:12:44.411121 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.411074 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjzg5\" (UniqueName: \"kubernetes.io/projected/d592f3e0-53bd-4928-a47a-f53d54c6600d-kube-api-access-zjzg5\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:12:44.411121 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.411086 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lkgx\" (UniqueName: \"kubernetes.io/projected/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-kube-api-access-5lkgx\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:12:44.411121 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.411095 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d592f3e0-53bd-4928-a47a-f53d54c6600d-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:12:44.411121 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.411104 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-openshift-service-ca\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:12:44.411121 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.411112 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb-metrics-certs\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:12:44.829153 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.829113 2576 generic.go:356] "Generic (PLEG): container finished" podID="63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb" containerID="a2521b2b779edac51c360cd2370de15c960df70ff1c4d98c28267a947cb34e8e" exitCode=0 Sep 05 08:12:44.829659 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.829163 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" Sep 05 08:12:44.829659 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.829199 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" event={"ID":"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb","Type":"ContainerDied","Data":"a2521b2b779edac51c360cd2370de15c960df70ff1c4d98c28267a947cb34e8e"} Sep 05 08:12:44.829659 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.829283 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h" event={"ID":"63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb","Type":"ContainerDied","Data":"ed91a56c9f03d34b17dc0d1dc2a9cf9f1bf70934e536ff34617483a0b0820ab8"} Sep 05 08:12:44.829659 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.829317 2576 scope.go:117] "RemoveContainer" containerID="a2521b2b779edac51c360cd2370de15c960df70ff1c4d98c28267a947cb34e8e" Sep 05 08:12:44.831034 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.831013 2576 generic.go:356] "Generic (PLEG): container finished" podID="d592f3e0-53bd-4928-a47a-f53d54c6600d" containerID="68d4b1a3f33824d21bc2bb208015aa3b5fec5c7c89ce45a5f42ff6cae28673f8" exitCode=0 Sep 05 08:12:44.831142 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.831125 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" event={"ID":"d592f3e0-53bd-4928-a47a-f53d54c6600d","Type":"ContainerDied","Data":"68d4b1a3f33824d21bc2bb208015aa3b5fec5c7c89ce45a5f42ff6cae28673f8"} Sep 05 08:12:44.831282 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.831153 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" Sep 05 08:12:44.831282 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.831173 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx" event={"ID":"d592f3e0-53bd-4928-a47a-f53d54c6600d","Type":"ContainerDied","Data":"b079e00432c02dbefe8178505ff3ef652d9e695244a06649475e8e6d8844207c"} Sep 05 08:12:44.832748 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.832724 2576 generic.go:356] "Generic (PLEG): container finished" podID="cb0ed63c-a287-4d29-999f-0d6c9c950064" containerID="e23ca6bce8562dcfb634b02b3e059d0cb87717068042e6a5c63be86b53a94921" exitCode=0 Sep 05 08:12:44.832879 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.832759 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" event={"ID":"cb0ed63c-a287-4d29-999f-0d6c9c950064","Type":"ContainerDied","Data":"e23ca6bce8562dcfb634b02b3e059d0cb87717068042e6a5c63be86b53a94921"} Sep 05 08:12:44.832879 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.832795 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" event={"ID":"cb0ed63c-a287-4d29-999f-0d6c9c950064","Type":"ContainerDied","Data":"5f4efb974fe26e05afbbe9ed2189c0533e125567476fb7bd35efc244d425ec4b"} Sep 05 08:12:44.832879 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.832812 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml" Sep 05 08:12:44.857506 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.857481 2576 scope.go:117] "RemoveContainer" containerID="a2521b2b779edac51c360cd2370de15c960df70ff1c4d98c28267a947cb34e8e" Sep 05 08:12:44.857834 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:12:44.857814 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2521b2b779edac51c360cd2370de15c960df70ff1c4d98c28267a947cb34e8e\": container with ID starting with a2521b2b779edac51c360cd2370de15c960df70ff1c4d98c28267a947cb34e8e not found: ID does not exist" containerID="a2521b2b779edac51c360cd2370de15c960df70ff1c4d98c28267a947cb34e8e" Sep 05 08:12:44.857897 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.857843 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2521b2b779edac51c360cd2370de15c960df70ff1c4d98c28267a947cb34e8e"} err="failed to get container status \"a2521b2b779edac51c360cd2370de15c960df70ff1c4d98c28267a947cb34e8e\": rpc error: code = NotFound desc = could not find container \"a2521b2b779edac51c360cd2370de15c960df70ff1c4d98c28267a947cb34e8e\": container with ID starting with a2521b2b779edac51c360cd2370de15c960df70ff1c4d98c28267a947cb34e8e not found: ID does not exist" Sep 05 08:12:44.857897 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.857860 2576 scope.go:117] "RemoveContainer" containerID="68d4b1a3f33824d21bc2bb208015aa3b5fec5c7c89ce45a5f42ff6cae28673f8" Sep 05 08:12:44.866642 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.866622 2576 scope.go:117] "RemoveContainer" containerID="68d4b1a3f33824d21bc2bb208015aa3b5fec5c7c89ce45a5f42ff6cae28673f8" Sep 05 08:12:44.866887 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:12:44.866865 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d4b1a3f33824d21bc2bb208015aa3b5fec5c7c89ce45a5f42ff6cae28673f8\": container with ID starting with 68d4b1a3f33824d21bc2bb208015aa3b5fec5c7c89ce45a5f42ff6cae28673f8 not found: ID does not exist" containerID="68d4b1a3f33824d21bc2bb208015aa3b5fec5c7c89ce45a5f42ff6cae28673f8" Sep 05 08:12:44.866936 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.866894 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d4b1a3f33824d21bc2bb208015aa3b5fec5c7c89ce45a5f42ff6cae28673f8"} err="failed to get container status \"68d4b1a3f33824d21bc2bb208015aa3b5fec5c7c89ce45a5f42ff6cae28673f8\": rpc error: code = NotFound desc = could not find container \"68d4b1a3f33824d21bc2bb208015aa3b5fec5c7c89ce45a5f42ff6cae28673f8\": container with ID starting with 68d4b1a3f33824d21bc2bb208015aa3b5fec5c7c89ce45a5f42ff6cae28673f8 not found: ID does not exist" Sep 05 08:12:44.866936 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.866910 2576 scope.go:117] "RemoveContainer" containerID="e23ca6bce8562dcfb634b02b3e059d0cb87717068042e6a5c63be86b53a94921" Sep 05 08:12:44.870908 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.870887 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx"] Sep 05 08:12:44.873480 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.873456 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8b6928-586675ddf9-mfhcx"] Sep 05 08:12:44.876297 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.876279 2576 scope.go:117] "RemoveContainer" containerID="e23ca6bce8562dcfb634b02b3e059d0cb87717068042e6a5c63be86b53a94921" Sep 05 08:12:44.876578 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:12:44.876557 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e23ca6bce8562dcfb634b02b3e059d0cb87717068042e6a5c63be86b53a94921\": container with ID starting with e23ca6bce8562dcfb634b02b3e059d0cb87717068042e6a5c63be86b53a94921 not found: ID does not exist" containerID="e23ca6bce8562dcfb634b02b3e059d0cb87717068042e6a5c63be86b53a94921" Sep 05 08:12:44.876653 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.876587 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e23ca6bce8562dcfb634b02b3e059d0cb87717068042e6a5c63be86b53a94921"} err="failed to get container status \"e23ca6bce8562dcfb634b02b3e059d0cb87717068042e6a5c63be86b53a94921\": rpc error: code = NotFound desc = could not find container \"e23ca6bce8562dcfb634b02b3e059d0cb87717068042e6a5c63be86b53a94921\": container with ID starting with e23ca6bce8562dcfb634b02b3e059d0cb87717068042e6a5c63be86b53a94921 not found: ID does not exist" Sep 05 08:12:44.882067 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.882046 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h"] Sep 05 08:12:44.888545 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.888525 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8b6928-5d954566d7-5ht5h"] Sep 05 08:12:44.897030 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.897006 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml"] Sep 05 08:12:44.902328 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:44.902301 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8b6928-784b6f6b58-nndml"] Sep 05 08:12:45.885826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:45.885791 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb" path="/var/lib/kubelet/pods/63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb/volumes" Sep 05 08:12:45.886233 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:45.886140 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb0ed63c-a287-4d29-999f-0d6c9c950064" path="/var/lib/kubelet/pods/cb0ed63c-a287-4d29-999f-0d6c9c950064/volumes" Sep 05 08:12:45.886472 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:45.886459 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d592f3e0-53bd-4928-a47a-f53d54c6600d" path="/var/lib/kubelet/pods/d592f3e0-53bd-4928-a47a-f53d54c6600d/volumes" Sep 05 08:12:50.043379 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:12:50.043336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:13:00.968283 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:00.968251 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn"] Sep 05 08:13:00.968794 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:00.968514 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" podUID="3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35" containerName="payload-processing" containerID="cri-o://57ea13c084c9f5a2012caf0e5007777f7e700133b76b3c09f541eb7007c39fd4" gracePeriod=30 Sep 05 08:13:00.973960 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:00.973914 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r"] Sep 05 08:13:00.974161 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:00.974134 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" podUID="8b546999-3cc4-4cc4-9ab2-d70470494ba8" containerName="payload-pre-processing" containerID="cri-o://37de408eca57e7f3c146c932818779b8623af7f4f82576dae9af21114464e257" gracePeriod=30 Sep 05 08:13:01.246686 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.246662 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:13:01.261082 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.261053 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" Sep 05 08:13:01.264064 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.264043 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" Sep 05 08:13:01.356650 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.356621 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxmtj\" (UniqueName: \"kubernetes.io/projected/8b546999-3cc4-4cc4-9ab2-d70470494ba8-kube-api-access-jxmtj\") pod \"8b546999-3cc4-4cc4-9ab2-d70470494ba8\" (UID: \"8b546999-3cc4-4cc4-9ab2-d70470494ba8\") " Sep 05 08:13:01.356829 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.356663 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35-plugins-config-volume\") pod \"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35\" (UID: \"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35\") " Sep 05 08:13:01.356829 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.356698 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn7sv\" (UniqueName: \"kubernetes.io/projected/3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35-kube-api-access-rn7sv\") pod \"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35\" (UID: \"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35\") " Sep 05 08:13:01.356829 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.356783 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b546999-3cc4-4cc4-9ab2-d70470494ba8-plugins-config-volume\") pod \"8b546999-3cc4-4cc4-9ab2-d70470494ba8\" (UID: \"8b546999-3cc4-4cc4-9ab2-d70470494ba8\") " Sep 05 08:13:01.357136 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.357106 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35" (UID: "3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:13:01.357261 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.357205 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b546999-3cc4-4cc4-9ab2-d70470494ba8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b546999-3cc4-4cc4-9ab2-d70470494ba8" (UID: "8b546999-3cc4-4cc4-9ab2-d70470494ba8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:13:01.359107 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.359084 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35-kube-api-access-rn7sv" (OuterVolumeSpecName: "kube-api-access-rn7sv") pod "3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35" (UID: "3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35"). InnerVolumeSpecName "kube-api-access-rn7sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:13:01.359107 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.359096 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b546999-3cc4-4cc4-9ab2-d70470494ba8-kube-api-access-jxmtj" (OuterVolumeSpecName: "kube-api-access-jxmtj") pod "8b546999-3cc4-4cc4-9ab2-d70470494ba8" (UID: "8b546999-3cc4-4cc4-9ab2-d70470494ba8"). InnerVolumeSpecName "kube-api-access-jxmtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:13:01.457552 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.457512 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b546999-3cc4-4cc4-9ab2-d70470494ba8-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:13:01.457552 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.457543 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxmtj\" (UniqueName: \"kubernetes.io/projected/8b546999-3cc4-4cc4-9ab2-d70470494ba8-kube-api-access-jxmtj\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:13:01.457552 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.457553 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:13:01.457552 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.457561 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn7sv\" (UniqueName: \"kubernetes.io/projected/3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35-kube-api-access-rn7sv\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:13:01.935105 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.935072 2576 generic.go:356] "Generic (PLEG): container finished" podID="3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35" containerID="57ea13c084c9f5a2012caf0e5007777f7e700133b76b3c09f541eb7007c39fd4" exitCode=0 Sep 05 08:13:01.935310 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.935118 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" Sep 05 08:13:01.935310 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.935153 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" event={"ID":"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35","Type":"ContainerDied","Data":"57ea13c084c9f5a2012caf0e5007777f7e700133b76b3c09f541eb7007c39fd4"} Sep 05 08:13:01.935310 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.935203 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn" event={"ID":"3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35","Type":"ContainerDied","Data":"f2e08d2c2b53e3bd61aa6fed374ee0e3e8d94902a9896b8b128c7fa489b923bb"} Sep 05 08:13:01.935310 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.935250 2576 scope.go:117] "RemoveContainer" containerID="57ea13c084c9f5a2012caf0e5007777f7e700133b76b3c09f541eb7007c39fd4" Sep 05 08:13:01.936625 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.936601 2576 generic.go:356] "Generic (PLEG): container finished" podID="8b546999-3cc4-4cc4-9ab2-d70470494ba8" containerID="37de408eca57e7f3c146c932818779b8623af7f4f82576dae9af21114464e257" exitCode=0 Sep 05 08:13:01.936738 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.936633 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" event={"ID":"8b546999-3cc4-4cc4-9ab2-d70470494ba8","Type":"ContainerDied","Data":"37de408eca57e7f3c146c932818779b8623af7f4f82576dae9af21114464e257"} Sep 05 08:13:01.936738 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.936687 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" event={"ID":"8b546999-3cc4-4cc4-9ab2-d70470494ba8","Type":"ContainerDied","Data":"4b1cff6dc765c5e12e5369484f8d9d76158de2a4285e07a5584c4f220d49b98b"} Sep 05 08:13:01.936738 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.936646 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r" Sep 05 08:13:01.944432 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.944415 2576 scope.go:117] "RemoveContainer" containerID="57ea13c084c9f5a2012caf0e5007777f7e700133b76b3c09f541eb7007c39fd4" Sep 05 08:13:01.944734 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:13:01.944710 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57ea13c084c9f5a2012caf0e5007777f7e700133b76b3c09f541eb7007c39fd4\": container with ID starting with 57ea13c084c9f5a2012caf0e5007777f7e700133b76b3c09f541eb7007c39fd4 not found: ID does not exist" containerID="57ea13c084c9f5a2012caf0e5007777f7e700133b76b3c09f541eb7007c39fd4" Sep 05 08:13:01.944824 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.944739 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57ea13c084c9f5a2012caf0e5007777f7e700133b76b3c09f541eb7007c39fd4"} err="failed to get container status \"57ea13c084c9f5a2012caf0e5007777f7e700133b76b3c09f541eb7007c39fd4\": rpc error: code = NotFound desc = could not find container \"57ea13c084c9f5a2012caf0e5007777f7e700133b76b3c09f541eb7007c39fd4\": container with ID starting with 57ea13c084c9f5a2012caf0e5007777f7e700133b76b3c09f541eb7007c39fd4 not found: ID does not exist" Sep 05 08:13:01.944824 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.944755 2576 scope.go:117] "RemoveContainer" containerID="37de408eca57e7f3c146c932818779b8623af7f4f82576dae9af21114464e257" Sep 05 08:13:01.952075 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.952052 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn"] Sep 05 08:13:01.953242 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.953224 2576 scope.go:117] "RemoveContainer" containerID="37de408eca57e7f3c146c932818779b8623af7f4f82576dae9af21114464e257" Sep 05 08:13:01.953583 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:13:01.953559 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37de408eca57e7f3c146c932818779b8623af7f4f82576dae9af21114464e257\": container with ID starting with 37de408eca57e7f3c146c932818779b8623af7f4f82576dae9af21114464e257 not found: ID does not exist" containerID="37de408eca57e7f3c146c932818779b8623af7f4f82576dae9af21114464e257" Sep 05 08:13:01.953715 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.953591 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37de408eca57e7f3c146c932818779b8623af7f4f82576dae9af21114464e257"} err="failed to get container status \"37de408eca57e7f3c146c932818779b8623af7f4f82576dae9af21114464e257\": rpc error: code = NotFound desc = could not find container \"37de408eca57e7f3c146c932818779b8623af7f4f82576dae9af21114464e257\": container with ID starting with 37de408eca57e7f3c146c932818779b8623af7f4f82576dae9af21114464e257 not found: ID does not exist" Sep 05 08:13:01.956261 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.956242 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-cec446-6bbdc87865-c4srn"] Sep 05 08:13:01.965734 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.965713 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r"] Sep 05 08:13:01.969899 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:01.969877 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-cec446-5cb9c958f4-lxq9r"] Sep 05 08:13:03.892502 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:03.892469 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35" path="/var/lib/kubelet/pods/3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35/volumes" Sep 05 08:13:03.892863 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:03.892813 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b546999-3cc4-4cc4-9ab2-d70470494ba8" path="/var/lib/kubelet/pods/8b546999-3cc4-4cc4-9ab2-d70470494ba8/volumes" Sep 05 08:13:14.545426 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:14.545385 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:13:17.978361 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:17.978327 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 08:13:17.979659 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:17.979638 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 08:13:17.981961 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:17.981934 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 08:13:17.983273 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:17.983257 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 08:13:23.938186 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938154 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz"] Sep 05 08:13:23.938592 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938540 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb0ed63c-a287-4d29-999f-0d6c9c950064" containerName="payload-pre-processing" Sep 05 08:13:23.938592 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938556 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb0ed63c-a287-4d29-999f-0d6c9c950064" containerName="payload-pre-processing" Sep 05 08:13:23.938592 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938566 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2375855-4588-4d7e-b3e6-bfb61a9f9342" containerName="payload-processing" Sep 05 08:13:23.938592 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938572 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2375855-4588-4d7e-b3e6-bfb61a9f9342" containerName="payload-processing" Sep 05 08:13:23.938592 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938582 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b546999-3cc4-4cc4-9ab2-d70470494ba8" containerName="payload-pre-processing" Sep 05 08:13:23.938592 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938587 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b546999-3cc4-4cc4-9ab2-d70470494ba8" containerName="payload-pre-processing" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938597 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb" containerName="maas-api" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938603 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb" containerName="maas-api" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938610 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9e11bd-bd86-45ee-a638-d5b834b25955" containerName="payload-processing" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938619 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9e11bd-bd86-45ee-a638-d5b834b25955" containerName="payload-processing" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938633 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d592f3e0-53bd-4928-a47a-f53d54c6600d" containerName="payload-processing" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938638 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d592f3e0-53bd-4928-a47a-f53d54c6600d" containerName="payload-processing" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938645 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35" containerName="payload-processing" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938650 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35" containerName="payload-processing" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938707 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ccfe075-4ca3-4ca3-a2a1-dcf5d0837c35" containerName="payload-processing" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938715 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a9e11bd-bd86-45ee-a638-d5b834b25955" containerName="payload-processing" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938721 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b546999-3cc4-4cc4-9ab2-d70470494ba8" containerName="payload-pre-processing" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938727 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb0ed63c-a287-4d29-999f-0d6c9c950064" containerName="payload-pre-processing" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938732 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d592f3e0-53bd-4928-a47a-f53d54c6600d" containerName="payload-processing" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938737 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2375855-4588-4d7e-b3e6-bfb61a9f9342" containerName="payload-processing" Sep 05 08:13:23.938792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.938742 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="63b615b9-5ab3-4aeb-aac8-8f4cb35fb2eb" containerName="maas-api" Sep 05 08:13:23.943285 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.943260 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:23.945401 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.945378 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3d4dc1-serving-cert\"" Sep 05 08:13:23.954968 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:23.954943 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz"] Sep 05 08:13:24.055280 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.055231 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-metrics-certs\") pod \"maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.055482 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.055324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.055482 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.055366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr2hf\" (UniqueName: \"kubernetes.io/projected/361d498d-5224-4661-9767-81f305eedf6b-kube-api-access-mr2hf\") pod \"maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.055612 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.055483 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/361d498d-5224-4661-9767-81f305eedf6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.156941 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.156905 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/361d498d-5224-4661-9767-81f305eedf6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.157129 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.156975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-metrics-certs\") pod \"maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.157129 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.157004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.157129 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.157023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr2hf\" (UniqueName: \"kubernetes.io/projected/361d498d-5224-4661-9767-81f305eedf6b-kube-api-access-mr2hf\") pod \"maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.157396 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:13:24.157153 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-serving-cert: secret "maas-api-e2e-ipp-a-3d4dc1-serving-cert" not found Sep 05 08:13:24.157396 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:13:24.157264 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-maas-api-tls podName:361d498d-5224-4661-9767-81f305eedf6b nodeName:}" failed. No retries permitted until 2026-09-05 08:13:24.657240463 +0000 UTC m=+1507.266215568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-maas-api-tls") pod "maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" (UID: "361d498d-5224-4661-9767-81f305eedf6b") : secret "maas-api-e2e-ipp-a-3d4dc1-serving-cert" not found Sep 05 08:13:24.157740 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.157713 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/361d498d-5224-4661-9767-81f305eedf6b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.159932 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.159905 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-metrics-certs\") pod \"maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.165951 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.165928 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr2hf\" (UniqueName: \"kubernetes.io/projected/361d498d-5224-4661-9767-81f305eedf6b-kube-api-access-mr2hf\") pod \"maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.602863 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.602818 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t"] Sep 05 08:13:24.606402 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.606378 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" Sep 05 08:13:24.608727 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.608681 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3d4dc1-dockercfg-f9twj\"" Sep 05 08:13:24.608895 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.608775 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3d4dc1\"" Sep 05 08:13:24.616275 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.616252 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq"] Sep 05 08:13:24.620548 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.620526 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t"] Sep 05 08:13:24.620832 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.620754 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" Sep 05 08:13:24.633066 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.633042 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq"] Sep 05 08:13:24.661538 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.661507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.664068 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.664048 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.762690 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.762649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd11bb8d-6247-4503-8060-245e0dc7b769-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq\" (UID: \"fd11bb8d-6247-4503-8060-245e0dc7b769\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" Sep 05 08:13:24.762893 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.762696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k62cc\" (UniqueName: \"kubernetes.io/projected/09c92d84-96c9-4747-8d3e-03e5f937d0da-kube-api-access-k62cc\") pod \"payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t\" (UID: \"09c92d84-96c9-4747-8d3e-03e5f937d0da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" Sep 05 08:13:24.762893 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.762767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhx5d\" (UniqueName: \"kubernetes.io/projected/fd11bb8d-6247-4503-8060-245e0dc7b769-kube-api-access-bhx5d\") pod \"payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq\" (UID: \"fd11bb8d-6247-4503-8060-245e0dc7b769\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" Sep 05 08:13:24.762893 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.762865 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c92d84-96c9-4747-8d3e-03e5f937d0da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t\" (UID: \"09c92d84-96c9-4747-8d3e-03e5f937d0da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" Sep 05 08:13:24.856371 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.856274 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:24.863724 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.863695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c92d84-96c9-4747-8d3e-03e5f937d0da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t\" (UID: \"09c92d84-96c9-4747-8d3e-03e5f937d0da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" Sep 05 08:13:24.863858 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.863752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd11bb8d-6247-4503-8060-245e0dc7b769-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq\" (UID: \"fd11bb8d-6247-4503-8060-245e0dc7b769\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" Sep 05 08:13:24.863858 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.863771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k62cc\" (UniqueName: \"kubernetes.io/projected/09c92d84-96c9-4747-8d3e-03e5f937d0da-kube-api-access-k62cc\") pod \"payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t\" (UID: \"09c92d84-96c9-4747-8d3e-03e5f937d0da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" Sep 05 08:13:24.863858 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.863797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhx5d\" (UniqueName: \"kubernetes.io/projected/fd11bb8d-6247-4503-8060-245e0dc7b769-kube-api-access-bhx5d\") pod \"payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq\" (UID: \"fd11bb8d-6247-4503-8060-245e0dc7b769\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" Sep 05 08:13:24.864497 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.864475 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd11bb8d-6247-4503-8060-245e0dc7b769-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq\" (UID: \"fd11bb8d-6247-4503-8060-245e0dc7b769\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" Sep 05 08:13:24.864632 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.864608 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c92d84-96c9-4747-8d3e-03e5f937d0da-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t\" (UID: \"09c92d84-96c9-4747-8d3e-03e5f937d0da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" Sep 05 08:13:24.872479 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.872306 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhx5d\" (UniqueName: \"kubernetes.io/projected/fd11bb8d-6247-4503-8060-245e0dc7b769-kube-api-access-bhx5d\") pod \"payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq\" (UID: \"fd11bb8d-6247-4503-8060-245e0dc7b769\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" Sep 05 08:13:24.872479 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.872413 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k62cc\" (UniqueName: \"kubernetes.io/projected/09c92d84-96c9-4747-8d3e-03e5f937d0da-kube-api-access-k62cc\") pod \"payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t\" (UID: \"09c92d84-96c9-4747-8d3e-03e5f937d0da\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" Sep 05 08:13:24.921769 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.921739 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" Sep 05 08:13:24.933434 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:24.933409 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" Sep 05 08:13:25.135640 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:25.135604 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz"] Sep 05 08:13:25.137658 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:13:25.137612 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361d498d_5224_4661_9767_81f305eedf6b.slice/crio-4498f3eb30225f0da97afabe979b4ada077729107dcf75c26b56c4827270de18 WatchSource:0}: Error finding container 4498f3eb30225f0da97afabe979b4ada077729107dcf75c26b56c4827270de18: Status 404 returned error can't find the container with id 4498f3eb30225f0da97afabe979b4ada077729107dcf75c26b56c4827270de18 Sep 05 08:13:25.138964 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:25.138941 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 08:13:25.199998 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:25.199967 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t"] Sep 05 08:13:25.201683 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:13:25.201656 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c92d84_96c9_4747_8d3e_03e5f937d0da.slice/crio-5e4252007b89b71c6125e5673dfacc6ecf34bb7d80cc062eeb6ee812407397c2 WatchSource:0}: Error finding container 5e4252007b89b71c6125e5673dfacc6ecf34bb7d80cc062eeb6ee812407397c2: Status 404 returned error can't find the container with id 5e4252007b89b71c6125e5673dfacc6ecf34bb7d80cc062eeb6ee812407397c2 Sep 05 08:13:25.212619 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:25.212596 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq"] Sep 05 08:13:25.216904 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:13:25.216879 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd11bb8d_6247_4503_8060_245e0dc7b769.slice/crio-3b6eb9fe8672594be9581f987354567c1db8d793b0cbbdab481e6c684c6b4585 WatchSource:0}: Error finding container 3b6eb9fe8672594be9581f987354567c1db8d793b0cbbdab481e6c684c6b4585: Status 404 returned error can't find the container with id 3b6eb9fe8672594be9581f987354567c1db8d793b0cbbdab481e6c684c6b4585 Sep 05 08:13:26.073720 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:26.073683 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" event={"ID":"361d498d-5224-4661-9767-81f305eedf6b","Type":"ContainerStarted","Data":"9f1bf5317cb6de1e7b01bf3c92b3a3c44c60d250d1ffb74209bed6a3c3744d80"} Sep 05 08:13:26.073925 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:26.073866 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:26.073925 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:26.073907 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" event={"ID":"361d498d-5224-4661-9767-81f305eedf6b","Type":"ContainerStarted","Data":"4498f3eb30225f0da97afabe979b4ada077729107dcf75c26b56c4827270de18"} Sep 05 08:13:26.075284 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:26.075258 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" event={"ID":"fd11bb8d-6247-4503-8060-245e0dc7b769","Type":"ContainerStarted","Data":"a275461d6d231988bc9ed2d9272ec20c5308210d6ea4756d2e7979c51669da06"} Sep 05 08:13:26.075443 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:26.075302 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" event={"ID":"fd11bb8d-6247-4503-8060-245e0dc7b769","Type":"ContainerStarted","Data":"3b6eb9fe8672594be9581f987354567c1db8d793b0cbbdab481e6c684c6b4585"} Sep 05 08:13:26.075443 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:26.075329 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" Sep 05 08:13:26.076918 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:26.076875 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" event={"ID":"09c92d84-96c9-4747-8d3e-03e5f937d0da","Type":"ContainerStarted","Data":"964143de1f8de3ee1bb2770adf5162b9f8265852bed7bb5fbea7368efab18854"} Sep 05 08:13:26.077058 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:26.076923 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" event={"ID":"09c92d84-96c9-4747-8d3e-03e5f937d0da","Type":"ContainerStarted","Data":"5e4252007b89b71c6125e5673dfacc6ecf34bb7d80cc062eeb6ee812407397c2"} Sep 05 08:13:26.077058 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:26.076973 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" Sep 05 08:13:26.089191 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:26.089135 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" podStartSLOduration=2.567358639 podStartE2EDuration="3.089118861s" totalImagesPullingTime="521.760222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:13:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:13:26.087942203 +0000 UTC m=+1508.696917327" watchObservedRunningTime="2026-09-05 08:13:26.089118861 +0000 UTC m=+1508.698093987" Sep 05 08:13:26.104955 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:26.104904 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" podStartSLOduration=2.104889302 podStartE2EDuration="2.104889302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:13:26.102743431 +0000 UTC m=+1508.711718554" watchObservedRunningTime="2026-09-05 08:13:26.104889302 +0000 UTC m=+1508.713864425" Sep 05 08:13:26.118770 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:26.118713 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" podStartSLOduration=2.118686516 podStartE2EDuration="2.118686516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:13:26.116538537 +0000 UTC m=+1508.725513660" watchObservedRunningTime="2026-09-05 08:13:26.118686516 +0000 UTC m=+1508.727661645" Sep 05 08:13:27.458700 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:27.458662 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:13:32.088875 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:32.088845 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:13:37.085630 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:37.085603 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" Sep 05 08:13:37.088327 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:37.085675 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" Sep 05 08:13:58.257059 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:13:58.257020 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:14:40.341101 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:14:40.341066 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:14:55.539532 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:14:55.539486 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:15:00.136617 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:00.136583 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3d4dc1-29809935-5kl99"] Sep 05 08:15:00.139937 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:00.139920 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8c2e20-29809935-q7lk6"] Sep 05 08:15:00.140047 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:00.140028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3d4dc1-29809935-5kl99" Sep 05 08:15:00.141845 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:00.141823 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpmmp\"" Sep 05 08:15:00.143148 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:00.143129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8c2e20-29809935-q7lk6" Sep 05 08:15:00.150279 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:00.150202 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3d4dc1-29809935-5kl99"] Sep 05 08:15:00.153642 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:00.153540 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8c2e20-29809935-q7lk6"] Sep 05 08:15:00.154576 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:00.154412 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3d4dc1-29809935-5kl99" Sep 05 08:15:00.163061 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:00.162624 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8c2e20-29809935-q7lk6" Sep 05 08:15:00.438619 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:00.438584 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3d4dc1-29809935-5kl99"] Sep 05 08:15:00.440528 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:15:00.440502 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b4a3be_9640_4f19_be8e_c945c819cd07.slice/crio-d3bc1c5acd771420381197abc3947c9e4677b955e9246fc1634221cf0d30751b WatchSource:0}: Error finding container d3bc1c5acd771420381197abc3947c9e4677b955e9246fc1634221cf0d30751b: Status 404 returned error can't find the container with id d3bc1c5acd771420381197abc3947c9e4677b955e9246fc1634221cf0d30751b Sep 05 08:15:00.446772 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:00.446719 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8c2e20-29809935-q7lk6"] Sep 05 08:15:00.449113 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:15:00.449092 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de37b77_2b8c_47d2_b48a_582c40e25ac7.slice/crio-64f7bd78e8174dacfe1d4b2d76b6148309fac5d2d21fed9d3f465f73bc2e3238 WatchSource:0}: Error finding container 64f7bd78e8174dacfe1d4b2d76b6148309fac5d2d21fed9d3f465f73bc2e3238: Status 404 returned error can't find the container with id 64f7bd78e8174dacfe1d4b2d76b6148309fac5d2d21fed9d3f465f73bc2e3238 Sep 05 08:15:00.636274 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:00.636241 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8c2e20-29809935-q7lk6" event={"ID":"1de37b77-2b8c-47d2-b48a-582c40e25ac7","Type":"ContainerStarted","Data":"64f7bd78e8174dacfe1d4b2d76b6148309fac5d2d21fed9d3f465f73bc2e3238"} Sep 05 08:15:00.637375 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:00.637346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3d4dc1-29809935-5kl99" event={"ID":"55b4a3be-9640-4f19-be8e-c945c819cd07","Type":"ContainerStarted","Data":"d3bc1c5acd771420381197abc3947c9e4677b955e9246fc1634221cf0d30751b"} Sep 05 08:15:01.645060 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:01.645024 2576 generic.go:356] "Generic (PLEG): container finished" podID="1de37b77-2b8c-47d2-b48a-582c40e25ac7" containerID="799c1949036b2dc5893c1421190ddb06da2729b52256547df4b53b909b6b3b9d" exitCode=0 Sep 05 08:15:01.645534 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:01.645076 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8c2e20-29809935-q7lk6" event={"ID":"1de37b77-2b8c-47d2-b48a-582c40e25ac7","Type":"ContainerDied","Data":"799c1949036b2dc5893c1421190ddb06da2729b52256547df4b53b909b6b3b9d"} Sep 05 08:15:01.646863 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:01.646834 2576 generic.go:356] "Generic (PLEG): container finished" podID="55b4a3be-9640-4f19-be8e-c945c819cd07" containerID="15a9e459e384772a7be5235b74b145836be4aa48e60b157e30c7e1aa66c9da67" exitCode=0 Sep 05 08:15:01.646977 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:01.646876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3d4dc1-29809935-5kl99" event={"ID":"55b4a3be-9640-4f19-be8e-c945c819cd07","Type":"ContainerDied","Data":"15a9e459e384772a7be5235b74b145836be4aa48e60b157e30c7e1aa66c9da67"} Sep 05 08:15:02.771863 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:02.771832 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq_fd11bb8d-6247-4503-8060-245e0dc7b769/payload-processing/0.log" Sep 05 08:15:02.792419 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:02.792391 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3d4dc1-29809935-5kl99" Sep 05 08:15:02.834711 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:02.834686 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8c2e20-29809935-q7lk6" Sep 05 08:15:03.660406 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:03.660363 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8c2e20-29809935-q7lk6" Sep 05 08:15:03.660406 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:03.660383 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8c2e20-29809935-q7lk6" event={"ID":"1de37b77-2b8c-47d2-b48a-582c40e25ac7","Type":"ContainerDied","Data":"64f7bd78e8174dacfe1d4b2d76b6148309fac5d2d21fed9d3f465f73bc2e3238"} Sep 05 08:15:03.660627 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:03.660437 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64f7bd78e8174dacfe1d4b2d76b6148309fac5d2d21fed9d3f465f73bc2e3238" Sep 05 08:15:03.661775 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:03.661755 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3d4dc1-29809935-5kl99" Sep 05 08:15:03.661906 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:03.661800 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3d4dc1-29809935-5kl99" event={"ID":"55b4a3be-9640-4f19-be8e-c945c819cd07","Type":"ContainerDied","Data":"d3bc1c5acd771420381197abc3947c9e4677b955e9246fc1634221cf0d30751b"} Sep 05 08:15:03.661906 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:03.661838 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3bc1c5acd771420381197abc3947c9e4677b955e9246fc1634221cf0d30751b" Sep 05 08:15:07.524966 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.524935 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r"] Sep 05 08:15:07.525964 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.525929 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de37b77-2b8c-47d2-b48a-582c40e25ac7" containerName="cleanup" Sep 05 08:15:07.525964 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.525954 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de37b77-2b8c-47d2-b48a-582c40e25ac7" containerName="cleanup" Sep 05 08:15:07.526133 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.525983 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b4a3be-9640-4f19-be8e-c945c819cd07" containerName="cleanup" Sep 05 08:15:07.526133 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.525992 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b4a3be-9640-4f19-be8e-c945c819cd07" containerName="cleanup" Sep 05 08:15:07.526133 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.526127 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="55b4a3be-9640-4f19-be8e-c945c819cd07" containerName="cleanup" Sep 05 08:15:07.526317 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.526142 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de37b77-2b8c-47d2-b48a-582c40e25ac7" containerName="cleanup" Sep 05 08:15:07.529341 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.529322 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:07.531449 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.531428 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-d9d46f-serving-cert\"" Sep 05 08:15:07.539037 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.538948 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r"] Sep 05 08:15:07.690337 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.690305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/117279a5-7719-430d-8e7e-2b39a9fe84c6-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:07.690530 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.690341 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2rtm\" (UniqueName: \"kubernetes.io/projected/117279a5-7719-430d-8e7e-2b39a9fe84c6-kube-api-access-g2rtm\") pod \"maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:07.690530 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.690370 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117279a5-7719-430d-8e7e-2b39a9fe84c6-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:07.690530 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.690501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/117279a5-7719-430d-8e7e-2b39a9fe84c6-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:07.791247 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.791126 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/117279a5-7719-430d-8e7e-2b39a9fe84c6-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:07.791247 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.791235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/117279a5-7719-430d-8e7e-2b39a9fe84c6-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:07.791496 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.791258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2rtm\" (UniqueName: \"kubernetes.io/projected/117279a5-7719-430d-8e7e-2b39a9fe84c6-kube-api-access-g2rtm\") pod \"maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:07.791496 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.791278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117279a5-7719-430d-8e7e-2b39a9fe84c6-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:07.791858 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.791830 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/117279a5-7719-430d-8e7e-2b39a9fe84c6-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:07.793837 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.793815 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117279a5-7719-430d-8e7e-2b39a9fe84c6-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:07.793971 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.793950 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/117279a5-7719-430d-8e7e-2b39a9fe84c6-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:07.798907 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.798878 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2rtm\" (UniqueName: \"kubernetes.io/projected/117279a5-7719-430d-8e7e-2b39a9fe84c6-kube-api-access-g2rtm\") pod \"maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:07.842007 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:07.841979 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:08.151714 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.148022 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r"] Sep 05 08:15:08.153434 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:15:08.153408 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117279a5_7719_430d_8e7e_2b39a9fe84c6.slice/crio-dad440a8b1c26154843c61f82428d976bfd741671b9a7556492c169725c27552 WatchSource:0}: Error finding container dad440a8b1c26154843c61f82428d976bfd741671b9a7556492c169725c27552: Status 404 returned error can't find the container with id dad440a8b1c26154843c61f82428d976bfd741671b9a7556492c169725c27552 Sep 05 08:15:08.356083 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.356053 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v"] Sep 05 08:15:08.361046 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.361023 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" Sep 05 08:15:08.363538 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.363488 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-d9d46f-dockercfg-l6zbl\"" Sep 05 08:15:08.363538 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.363516 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-d9d46f\"" Sep 05 08:15:08.370763 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.370733 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v"] Sep 05 08:15:08.498106 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.498009 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dppkb\" (UniqueName: \"kubernetes.io/projected/36d06474-f9f5-45bd-b520-dc182f585ef3-kube-api-access-dppkb\") pod \"payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v\" (UID: \"36d06474-f9f5-45bd-b520-dc182f585ef3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" Sep 05 08:15:08.498106 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.498094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d06474-f9f5-45bd-b520-dc182f585ef3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v\" (UID: \"36d06474-f9f5-45bd-b520-dc182f585ef3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" Sep 05 08:15:08.598641 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.598615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d06474-f9f5-45bd-b520-dc182f585ef3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v\" (UID: \"36d06474-f9f5-45bd-b520-dc182f585ef3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" Sep 05 08:15:08.599030 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.598724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dppkb\" (UniqueName: \"kubernetes.io/projected/36d06474-f9f5-45bd-b520-dc182f585ef3-kube-api-access-dppkb\") pod \"payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v\" (UID: \"36d06474-f9f5-45bd-b520-dc182f585ef3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" Sep 05 08:15:08.599365 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.599346 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d06474-f9f5-45bd-b520-dc182f585ef3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v\" (UID: \"36d06474-f9f5-45bd-b520-dc182f585ef3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" Sep 05 08:15:08.606799 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.606780 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dppkb\" (UniqueName: \"kubernetes.io/projected/36d06474-f9f5-45bd-b520-dc182f585ef3-kube-api-access-dppkb\") pod \"payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v\" (UID: \"36d06474-f9f5-45bd-b520-dc182f585ef3\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" Sep 05 08:15:08.675996 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.675964 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" Sep 05 08:15:08.694764 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.694734 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" event={"ID":"117279a5-7719-430d-8e7e-2b39a9fe84c6","Type":"ContainerStarted","Data":"fd2895f8b24f65f87ca9928b6bc462e8434044b4b1e3158e97e49c32d2f93141"} Sep 05 08:15:08.694889 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.694794 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" event={"ID":"117279a5-7719-430d-8e7e-2b39a9fe84c6","Type":"ContainerStarted","Data":"dad440a8b1c26154843c61f82428d976bfd741671b9a7556492c169725c27552"} Sep 05 08:15:08.694972 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.694949 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:08.710836 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.710789 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" podStartSLOduration=1.324807182 podStartE2EDuration="1.710774419s" totalImagesPullingTime="385.967237ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:15:08.709499553 +0000 UTC m=+1611.318474680" watchObservedRunningTime="2026-09-05 08:15:08.710774419 +0000 UTC m=+1611.319749593" Sep 05 08:15:08.950020 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:08.949996 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v"] Sep 05 08:15:08.951920 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:15:08.951895 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d06474_f9f5_45bd_b520_dc182f585ef3.slice/crio-c9085f7820304c1007be303e73d2d091161a485e025a8a4a3f6bff6a75113e55 WatchSource:0}: Error finding container c9085f7820304c1007be303e73d2d091161a485e025a8a4a3f6bff6a75113e55: Status 404 returned error can't find the container with id c9085f7820304c1007be303e73d2d091161a485e025a8a4a3f6bff6a75113e55 Sep 05 08:15:09.703096 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:09.703064 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" event={"ID":"36d06474-f9f5-45bd-b520-dc182f585ef3","Type":"ContainerStarted","Data":"f8ac86c325ceb7a76a2d89a0a49b31654e8ba554035891acb5072da10b47d853"} Sep 05 08:15:09.703568 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:09.703117 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" event={"ID":"36d06474-f9f5-45bd-b520-dc182f585ef3","Type":"ContainerStarted","Data":"c9085f7820304c1007be303e73d2d091161a485e025a8a4a3f6bff6a75113e55"} Sep 05 08:15:09.703568 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:09.703297 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" Sep 05 08:15:09.717873 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:09.717824 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" podStartSLOduration=1.717808861 podStartE2EDuration="1.717808861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:15:09.717393405 +0000 UTC m=+1612.326368543" watchObservedRunningTime="2026-09-05 08:15:09.717808861 +0000 UTC m=+1612.326783992" Sep 05 08:15:14.708790 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:14.708756 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:15.041671 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:15.041565 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:15:20.710851 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:20.710819 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" Sep 05 08:15:38.108202 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.108164 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r"] Sep 05 08:15:38.108637 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.108436 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" podUID="117279a5-7719-430d-8e7e-2b39a9fe84c6" containerName="maas-api" containerID="cri-o://fd2895f8b24f65f87ca9928b6bc462e8434044b4b1e3158e97e49c32d2f93141" gracePeriod=30 Sep 05 08:15:38.158935 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.158900 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v"] Sep 05 08:15:38.159254 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.159223 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" podUID="36d06474-f9f5-45bd-b520-dc182f585ef3" containerName="payload-processing" containerID="cri-o://f8ac86c325ceb7a76a2d89a0a49b31654e8ba554035891acb5072da10b47d853" gracePeriod=30 Sep 05 08:15:38.447586 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.447553 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:38.451613 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.451580 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" Sep 05 08:15:38.579240 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.579182 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/117279a5-7719-430d-8e7e-2b39a9fe84c6-openshift-service-ca\") pod \"117279a5-7719-430d-8e7e-2b39a9fe84c6\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " Sep 05 08:15:38.579422 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.579255 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dppkb\" (UniqueName: \"kubernetes.io/projected/36d06474-f9f5-45bd-b520-dc182f585ef3-kube-api-access-dppkb\") pod \"36d06474-f9f5-45bd-b520-dc182f585ef3\" (UID: \"36d06474-f9f5-45bd-b520-dc182f585ef3\") " Sep 05 08:15:38.579422 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.579349 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/117279a5-7719-430d-8e7e-2b39a9fe84c6-maas-api-tls\") pod \"117279a5-7719-430d-8e7e-2b39a9fe84c6\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " Sep 05 08:15:38.579422 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.579400 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117279a5-7719-430d-8e7e-2b39a9fe84c6-metrics-certs\") pod \"117279a5-7719-430d-8e7e-2b39a9fe84c6\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " Sep 05 08:15:38.579615 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.579443 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d06474-f9f5-45bd-b520-dc182f585ef3-plugins-config-volume\") pod \"36d06474-f9f5-45bd-b520-dc182f585ef3\" (UID: \"36d06474-f9f5-45bd-b520-dc182f585ef3\") " Sep 05 08:15:38.579615 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.579515 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2rtm\" (UniqueName: \"kubernetes.io/projected/117279a5-7719-430d-8e7e-2b39a9fe84c6-kube-api-access-g2rtm\") pod \"117279a5-7719-430d-8e7e-2b39a9fe84c6\" (UID: \"117279a5-7719-430d-8e7e-2b39a9fe84c6\") " Sep 05 08:15:38.579615 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.579560 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117279a5-7719-430d-8e7e-2b39a9fe84c6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "117279a5-7719-430d-8e7e-2b39a9fe84c6" (UID: "117279a5-7719-430d-8e7e-2b39a9fe84c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:15:38.580869 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.580843 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/117279a5-7719-430d-8e7e-2b39a9fe84c6-openshift-service-ca\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:15:38.581820 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.581784 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d06474-f9f5-45bd-b520-dc182f585ef3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36d06474-f9f5-45bd-b520-dc182f585ef3" (UID: "36d06474-f9f5-45bd-b520-dc182f585ef3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:15:38.589951 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.589917 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/117279a5-7719-430d-8e7e-2b39a9fe84c6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "117279a5-7719-430d-8e7e-2b39a9fe84c6" (UID: "117279a5-7719-430d-8e7e-2b39a9fe84c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:15:38.589951 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.589925 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117279a5-7719-430d-8e7e-2b39a9fe84c6-kube-api-access-g2rtm" (OuterVolumeSpecName: "kube-api-access-g2rtm") pod "117279a5-7719-430d-8e7e-2b39a9fe84c6" (UID: "117279a5-7719-430d-8e7e-2b39a9fe84c6"). InnerVolumeSpecName "kube-api-access-g2rtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:15:38.590144 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.589951 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d06474-f9f5-45bd-b520-dc182f585ef3-kube-api-access-dppkb" (OuterVolumeSpecName: "kube-api-access-dppkb") pod "36d06474-f9f5-45bd-b520-dc182f585ef3" (UID: "36d06474-f9f5-45bd-b520-dc182f585ef3"). InnerVolumeSpecName "kube-api-access-dppkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:15:38.591652 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.591617 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/117279a5-7719-430d-8e7e-2b39a9fe84c6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "117279a5-7719-430d-8e7e-2b39a9fe84c6" (UID: "117279a5-7719-430d-8e7e-2b39a9fe84c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:15:38.681779 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.681682 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dppkb\" (UniqueName: \"kubernetes.io/projected/36d06474-f9f5-45bd-b520-dc182f585ef3-kube-api-access-dppkb\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:15:38.681779 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.681719 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/117279a5-7719-430d-8e7e-2b39a9fe84c6-maas-api-tls\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:15:38.681779 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.681729 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117279a5-7719-430d-8e7e-2b39a9fe84c6-metrics-certs\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:15:38.681779 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.681737 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d06474-f9f5-45bd-b520-dc182f585ef3-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:15:38.681779 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.681747 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2rtm\" (UniqueName: \"kubernetes.io/projected/117279a5-7719-430d-8e7e-2b39a9fe84c6-kube-api-access-g2rtm\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:15:38.871327 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.871288 2576 generic.go:356] "Generic (PLEG): container finished" podID="36d06474-f9f5-45bd-b520-dc182f585ef3" containerID="f8ac86c325ceb7a76a2d89a0a49b31654e8ba554035891acb5072da10b47d853" exitCode=0 Sep 05 08:15:38.871506 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.871346 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" Sep 05 08:15:38.871506 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.871371 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" event={"ID":"36d06474-f9f5-45bd-b520-dc182f585ef3","Type":"ContainerDied","Data":"f8ac86c325ceb7a76a2d89a0a49b31654e8ba554035891acb5072da10b47d853"} Sep 05 08:15:38.871506 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.871450 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v" event={"ID":"36d06474-f9f5-45bd-b520-dc182f585ef3","Type":"ContainerDied","Data":"c9085f7820304c1007be303e73d2d091161a485e025a8a4a3f6bff6a75113e55"} Sep 05 08:15:38.871506 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.871478 2576 scope.go:117] "RemoveContainer" containerID="f8ac86c325ceb7a76a2d89a0a49b31654e8ba554035891acb5072da10b47d853" Sep 05 08:15:38.873292 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.873257 2576 generic.go:356] "Generic (PLEG): container finished" podID="117279a5-7719-430d-8e7e-2b39a9fe84c6" containerID="fd2895f8b24f65f87ca9928b6bc462e8434044b4b1e3158e97e49c32d2f93141" exitCode=0 Sep 05 08:15:38.873437 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.873312 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" Sep 05 08:15:38.873437 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.873346 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" event={"ID":"117279a5-7719-430d-8e7e-2b39a9fe84c6","Type":"ContainerDied","Data":"fd2895f8b24f65f87ca9928b6bc462e8434044b4b1e3158e97e49c32d2f93141"} Sep 05 08:15:38.873437 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.873387 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r" event={"ID":"117279a5-7719-430d-8e7e-2b39a9fe84c6","Type":"ContainerDied","Data":"dad440a8b1c26154843c61f82428d976bfd741671b9a7556492c169725c27552"} Sep 05 08:15:38.881949 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.881837 2576 scope.go:117] "RemoveContainer" containerID="f8ac86c325ceb7a76a2d89a0a49b31654e8ba554035891acb5072da10b47d853" Sep 05 08:15:38.882187 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:15:38.882161 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ac86c325ceb7a76a2d89a0a49b31654e8ba554035891acb5072da10b47d853\": container with ID starting with f8ac86c325ceb7a76a2d89a0a49b31654e8ba554035891acb5072da10b47d853 not found: ID does not exist" containerID="f8ac86c325ceb7a76a2d89a0a49b31654e8ba554035891acb5072da10b47d853" Sep 05 08:15:38.882330 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.882197 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ac86c325ceb7a76a2d89a0a49b31654e8ba554035891acb5072da10b47d853"} err="failed to get container status \"f8ac86c325ceb7a76a2d89a0a49b31654e8ba554035891acb5072da10b47d853\": rpc error: code = NotFound desc = could not find container \"f8ac86c325ceb7a76a2d89a0a49b31654e8ba554035891acb5072da10b47d853\": container with ID starting with f8ac86c325ceb7a76a2d89a0a49b31654e8ba554035891acb5072da10b47d853 not found: ID does not exist" Sep 05 08:15:38.882330 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.882236 2576 scope.go:117] "RemoveContainer" containerID="fd2895f8b24f65f87ca9928b6bc462e8434044b4b1e3158e97e49c32d2f93141" Sep 05 08:15:38.890951 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.890930 2576 scope.go:117] "RemoveContainer" containerID="fd2895f8b24f65f87ca9928b6bc462e8434044b4b1e3158e97e49c32d2f93141" Sep 05 08:15:38.891294 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:15:38.891268 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2895f8b24f65f87ca9928b6bc462e8434044b4b1e3158e97e49c32d2f93141\": container with ID starting with fd2895f8b24f65f87ca9928b6bc462e8434044b4b1e3158e97e49c32d2f93141 not found: ID does not exist" containerID="fd2895f8b24f65f87ca9928b6bc462e8434044b4b1e3158e97e49c32d2f93141" Sep 05 08:15:38.891391 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.891299 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2895f8b24f65f87ca9928b6bc462e8434044b4b1e3158e97e49c32d2f93141"} err="failed to get container status \"fd2895f8b24f65f87ca9928b6bc462e8434044b4b1e3158e97e49c32d2f93141\": rpc error: code = NotFound desc = could not find container \"fd2895f8b24f65f87ca9928b6bc462e8434044b4b1e3158e97e49c32d2f93141\": container with ID starting with fd2895f8b24f65f87ca9928b6bc462e8434044b4b1e3158e97e49c32d2f93141 not found: ID does not exist" Sep 05 08:15:38.896132 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.896111 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r"] Sep 05 08:15:38.899746 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.899728 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d9d46f-ccdbf998d-d2v2r"] Sep 05 08:15:38.907205 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.907182 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v"] Sep 05 08:15:38.909458 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:38.909441 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-d9d46f-75994f9c-fvc9v"] Sep 05 08:15:39.886003 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:39.885971 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="117279a5-7719-430d-8e7e-2b39a9fe84c6" path="/var/lib/kubelet/pods/117279a5-7719-430d-8e7e-2b39a9fe84c6/volumes" Sep 05 08:15:39.886427 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:39.886380 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36d06474-f9f5-45bd-b520-dc182f585ef3" path="/var/lib/kubelet/pods/36d06474-f9f5-45bd-b520-dc182f585ef3/volumes" Sep 05 08:15:44.047485 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:15:44.047448 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:16:03.540515 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:03.540478 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:16:04.127719 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.127680 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz"] Sep 05 08:16:04.127994 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.127968 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" podUID="361d498d-5224-4661-9767-81f305eedf6b" containerName="maas-api" containerID="cri-o://9f1bf5317cb6de1e7b01bf3c92b3a3c44c60d250d1ffb74209bed6a3c3744d80" gracePeriod=30 Sep 05 08:16:04.153202 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.153165 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3d4dc1-29809935-5kl99"] Sep 05 08:16:04.158186 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.158155 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-3d4dc1-29809935-5kl99"] Sep 05 08:16:04.179003 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.178966 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq"] Sep 05 08:16:04.179320 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.179266 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" podUID="fd11bb8d-6247-4503-8060-245e0dc7b769" containerName="payload-processing" containerID="cri-o://a275461d6d231988bc9ed2d9272ec20c5308210d6ea4756d2e7979c51669da06" gracePeriod=30 Sep 05 08:16:04.185011 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.184982 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t"] Sep 05 08:16:04.185245 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.185199 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" podUID="09c92d84-96c9-4747-8d3e-03e5f937d0da" containerName="payload-pre-processing" containerID="cri-o://964143de1f8de3ee1bb2770adf5162b9f8265852bed7bb5fbea7368efab18854" gracePeriod=30 Sep 05 08:16:04.473774 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.473741 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:16:04.476993 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.476966 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" Sep 05 08:16:04.482383 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.482363 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" Sep 05 08:16:04.615497 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.615460 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd11bb8d-6247-4503-8060-245e0dc7b769-plugins-config-volume\") pod \"fd11bb8d-6247-4503-8060-245e0dc7b769\" (UID: \"fd11bb8d-6247-4503-8060-245e0dc7b769\") " Sep 05 08:16:04.616033 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.615509 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c92d84-96c9-4747-8d3e-03e5f937d0da-plugins-config-volume\") pod \"09c92d84-96c9-4747-8d3e-03e5f937d0da\" (UID: \"09c92d84-96c9-4747-8d3e-03e5f937d0da\") " Sep 05 08:16:04.616033 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.615556 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mr2hf\" (UniqueName: \"kubernetes.io/projected/361d498d-5224-4661-9767-81f305eedf6b-kube-api-access-mr2hf\") pod \"361d498d-5224-4661-9767-81f305eedf6b\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " Sep 05 08:16:04.616033 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.615575 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/361d498d-5224-4661-9767-81f305eedf6b-openshift-service-ca\") pod \"361d498d-5224-4661-9767-81f305eedf6b\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " Sep 05 08:16:04.616033 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.615607 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhx5d\" (UniqueName: \"kubernetes.io/projected/fd11bb8d-6247-4503-8060-245e0dc7b769-kube-api-access-bhx5d\") pod \"fd11bb8d-6247-4503-8060-245e0dc7b769\" (UID: \"fd11bb8d-6247-4503-8060-245e0dc7b769\") " Sep 05 08:16:04.616033 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.615644 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-maas-api-tls\") pod \"361d498d-5224-4661-9767-81f305eedf6b\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " Sep 05 08:16:04.616033 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.615926 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd11bb8d-6247-4503-8060-245e0dc7b769-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd11bb8d-6247-4503-8060-245e0dc7b769" (UID: "fd11bb8d-6247-4503-8060-245e0dc7b769"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:16:04.616033 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.615970 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/361d498d-5224-4661-9767-81f305eedf6b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "361d498d-5224-4661-9767-81f305eedf6b" (UID: "361d498d-5224-4661-9767-81f305eedf6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:16:04.616033 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.615985 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c92d84-96c9-4747-8d3e-03e5f937d0da-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09c92d84-96c9-4747-8d3e-03e5f937d0da" (UID: "09c92d84-96c9-4747-8d3e-03e5f937d0da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:16:04.616033 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.616025 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k62cc\" (UniqueName: \"kubernetes.io/projected/09c92d84-96c9-4747-8d3e-03e5f937d0da-kube-api-access-k62cc\") pod \"09c92d84-96c9-4747-8d3e-03e5f937d0da\" (UID: \"09c92d84-96c9-4747-8d3e-03e5f937d0da\") " Sep 05 08:16:04.616866 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.616055 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-metrics-certs\") pod \"361d498d-5224-4661-9767-81f305eedf6b\" (UID: \"361d498d-5224-4661-9767-81f305eedf6b\") " Sep 05 08:16:04.616866 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.616397 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd11bb8d-6247-4503-8060-245e0dc7b769-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:16:04.616866 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.616417 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09c92d84-96c9-4747-8d3e-03e5f937d0da-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:16:04.616866 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.616432 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/361d498d-5224-4661-9767-81f305eedf6b-openshift-service-ca\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:16:04.618280 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.618245 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd11bb8d-6247-4503-8060-245e0dc7b769-kube-api-access-bhx5d" (OuterVolumeSpecName: "kube-api-access-bhx5d") pod "fd11bb8d-6247-4503-8060-245e0dc7b769" (UID: "fd11bb8d-6247-4503-8060-245e0dc7b769"). InnerVolumeSpecName "kube-api-access-bhx5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:16:04.618405 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.618338 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/361d498d-5224-4661-9767-81f305eedf6b-kube-api-access-mr2hf" (OuterVolumeSpecName: "kube-api-access-mr2hf") pod "361d498d-5224-4661-9767-81f305eedf6b" (UID: "361d498d-5224-4661-9767-81f305eedf6b"). InnerVolumeSpecName "kube-api-access-mr2hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:16:04.618405 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.618393 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "361d498d-5224-4661-9767-81f305eedf6b" (UID: "361d498d-5224-4661-9767-81f305eedf6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:16:04.618900 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.618872 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "361d498d-5224-4661-9767-81f305eedf6b" (UID: "361d498d-5224-4661-9767-81f305eedf6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:16:04.619677 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.619647 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c92d84-96c9-4747-8d3e-03e5f937d0da-kube-api-access-k62cc" (OuterVolumeSpecName: "kube-api-access-k62cc") pod "09c92d84-96c9-4747-8d3e-03e5f937d0da" (UID: "09c92d84-96c9-4747-8d3e-03e5f937d0da"). InnerVolumeSpecName "kube-api-access-k62cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:16:04.717682 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.717463 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr2hf\" (UniqueName: \"kubernetes.io/projected/361d498d-5224-4661-9767-81f305eedf6b-kube-api-access-mr2hf\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:16:04.717682 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.717505 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhx5d\" (UniqueName: \"kubernetes.io/projected/fd11bb8d-6247-4503-8060-245e0dc7b769-kube-api-access-bhx5d\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:16:04.717682 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.717522 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-maas-api-tls\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:16:04.717682 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.717535 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k62cc\" (UniqueName: \"kubernetes.io/projected/09c92d84-96c9-4747-8d3e-03e5f937d0da-kube-api-access-k62cc\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:16:04.717682 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:04.717549 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/361d498d-5224-4661-9767-81f305eedf6b-metrics-certs\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:16:05.028792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.028688 2576 generic.go:356] "Generic (PLEG): container finished" podID="361d498d-5224-4661-9767-81f305eedf6b" containerID="9f1bf5317cb6de1e7b01bf3c92b3a3c44c60d250d1ffb74209bed6a3c3744d80" exitCode=0 Sep 05 08:16:05.028792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.028749 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" Sep 05 08:16:05.028792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.028778 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" event={"ID":"361d498d-5224-4661-9767-81f305eedf6b","Type":"ContainerDied","Data":"9f1bf5317cb6de1e7b01bf3c92b3a3c44c60d250d1ffb74209bed6a3c3744d80"} Sep 05 08:16:05.029087 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.028839 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz" event={"ID":"361d498d-5224-4661-9767-81f305eedf6b","Type":"ContainerDied","Data":"4498f3eb30225f0da97afabe979b4ada077729107dcf75c26b56c4827270de18"} Sep 05 08:16:05.029087 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.028869 2576 scope.go:117] "RemoveContainer" containerID="9f1bf5317cb6de1e7b01bf3c92b3a3c44c60d250d1ffb74209bed6a3c3744d80" Sep 05 08:16:05.030339 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.030314 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd11bb8d-6247-4503-8060-245e0dc7b769" containerID="a275461d6d231988bc9ed2d9272ec20c5308210d6ea4756d2e7979c51669da06" exitCode=0 Sep 05 08:16:05.030472 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.030355 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" Sep 05 08:16:05.030472 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.030376 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" event={"ID":"fd11bb8d-6247-4503-8060-245e0dc7b769","Type":"ContainerDied","Data":"a275461d6d231988bc9ed2d9272ec20c5308210d6ea4756d2e7979c51669da06"} Sep 05 08:16:05.030603 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.030590 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq" event={"ID":"fd11bb8d-6247-4503-8060-245e0dc7b769","Type":"ContainerDied","Data":"3b6eb9fe8672594be9581f987354567c1db8d793b0cbbdab481e6c684c6b4585"} Sep 05 08:16:05.031925 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.031898 2576 generic.go:356] "Generic (PLEG): container finished" podID="09c92d84-96c9-4747-8d3e-03e5f937d0da" containerID="964143de1f8de3ee1bb2770adf5162b9f8265852bed7bb5fbea7368efab18854" exitCode=0 Sep 05 08:16:05.032038 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.031932 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" Sep 05 08:16:05.032038 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.031958 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" event={"ID":"09c92d84-96c9-4747-8d3e-03e5f937d0da","Type":"ContainerDied","Data":"964143de1f8de3ee1bb2770adf5162b9f8265852bed7bb5fbea7368efab18854"} Sep 05 08:16:05.032038 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.032001 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t" event={"ID":"09c92d84-96c9-4747-8d3e-03e5f937d0da","Type":"ContainerDied","Data":"5e4252007b89b71c6125e5673dfacc6ecf34bb7d80cc062eeb6ee812407397c2"} Sep 05 08:16:05.040297 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.040274 2576 scope.go:117] "RemoveContainer" containerID="9f1bf5317cb6de1e7b01bf3c92b3a3c44c60d250d1ffb74209bed6a3c3744d80" Sep 05 08:16:05.040589 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:16:05.040571 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1bf5317cb6de1e7b01bf3c92b3a3c44c60d250d1ffb74209bed6a3c3744d80\": container with ID starting with 9f1bf5317cb6de1e7b01bf3c92b3a3c44c60d250d1ffb74209bed6a3c3744d80 not found: ID does not exist" containerID="9f1bf5317cb6de1e7b01bf3c92b3a3c44c60d250d1ffb74209bed6a3c3744d80" Sep 05 08:16:05.040686 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.040599 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1bf5317cb6de1e7b01bf3c92b3a3c44c60d250d1ffb74209bed6a3c3744d80"} err="failed to get container status \"9f1bf5317cb6de1e7b01bf3c92b3a3c44c60d250d1ffb74209bed6a3c3744d80\": rpc error: code = NotFound desc = could not find container \"9f1bf5317cb6de1e7b01bf3c92b3a3c44c60d250d1ffb74209bed6a3c3744d80\": container with ID starting with 9f1bf5317cb6de1e7b01bf3c92b3a3c44c60d250d1ffb74209bed6a3c3744d80 not found: ID does not exist" Sep 05 08:16:05.040686 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.040615 2576 scope.go:117] "RemoveContainer" containerID="a275461d6d231988bc9ed2d9272ec20c5308210d6ea4756d2e7979c51669da06" Sep 05 08:16:05.049700 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.049657 2576 scope.go:117] "RemoveContainer" containerID="a275461d6d231988bc9ed2d9272ec20c5308210d6ea4756d2e7979c51669da06" Sep 05 08:16:05.050286 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:16:05.050263 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a275461d6d231988bc9ed2d9272ec20c5308210d6ea4756d2e7979c51669da06\": container with ID starting with a275461d6d231988bc9ed2d9272ec20c5308210d6ea4756d2e7979c51669da06 not found: ID does not exist" containerID="a275461d6d231988bc9ed2d9272ec20c5308210d6ea4756d2e7979c51669da06" Sep 05 08:16:05.050388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.050291 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a275461d6d231988bc9ed2d9272ec20c5308210d6ea4756d2e7979c51669da06"} err="failed to get container status \"a275461d6d231988bc9ed2d9272ec20c5308210d6ea4756d2e7979c51669da06\": rpc error: code = NotFound desc = could not find container \"a275461d6d231988bc9ed2d9272ec20c5308210d6ea4756d2e7979c51669da06\": container with ID starting with a275461d6d231988bc9ed2d9272ec20c5308210d6ea4756d2e7979c51669da06 not found: ID does not exist" Sep 05 08:16:05.050388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.050308 2576 scope.go:117] "RemoveContainer" containerID="964143de1f8de3ee1bb2770adf5162b9f8265852bed7bb5fbea7368efab18854" Sep 05 08:16:05.052371 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.052346 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz"] Sep 05 08:16:05.059395 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.059376 2576 scope.go:117] "RemoveContainer" containerID="964143de1f8de3ee1bb2770adf5162b9f8265852bed7bb5fbea7368efab18854" Sep 05 08:16:05.059690 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:16:05.059672 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964143de1f8de3ee1bb2770adf5162b9f8265852bed7bb5fbea7368efab18854\": container with ID starting with 964143de1f8de3ee1bb2770adf5162b9f8265852bed7bb5fbea7368efab18854 not found: ID does not exist" containerID="964143de1f8de3ee1bb2770adf5162b9f8265852bed7bb5fbea7368efab18854" Sep 05 08:16:05.059763 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.059706 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964143de1f8de3ee1bb2770adf5162b9f8265852bed7bb5fbea7368efab18854"} err="failed to get container status \"964143de1f8de3ee1bb2770adf5162b9f8265852bed7bb5fbea7368efab18854\": rpc error: code = NotFound desc = could not find container \"964143de1f8de3ee1bb2770adf5162b9f8265852bed7bb5fbea7368efab18854\": container with ID starting with 964143de1f8de3ee1bb2770adf5162b9f8265852bed7bb5fbea7368efab18854 not found: ID does not exist" Sep 05 08:16:05.060903 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.060878 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3d4dc1-7bcf767655-nvrhz"] Sep 05 08:16:05.070125 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.070102 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq"] Sep 05 08:16:05.073698 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.073677 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3d4dc1-7b46775b99-chnwq"] Sep 05 08:16:05.082598 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.082575 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t"] Sep 05 08:16:05.086480 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.086458 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3d4dc1-695fb45fbb-m6k4t"] Sep 05 08:16:05.887019 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.886982 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09c92d84-96c9-4747-8d3e-03e5f937d0da" path="/var/lib/kubelet/pods/09c92d84-96c9-4747-8d3e-03e5f937d0da/volumes" Sep 05 08:16:05.887447 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.887432 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="361d498d-5224-4661-9767-81f305eedf6b" path="/var/lib/kubelet/pods/361d498d-5224-4661-9767-81f305eedf6b/volumes" Sep 05 08:16:05.887723 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.887710 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55b4a3be-9640-4f19-be8e-c945c819cd07" path="/var/lib/kubelet/pods/55b4a3be-9640-4f19-be8e-c945c819cd07/volumes" Sep 05 08:16:05.888024 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:05.888012 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd11bb8d-6247-4503-8060-245e0dc7b769" path="/var/lib/kubelet/pods/fd11bb8d-6247-4503-8060-245e0dc7b769/volumes" Sep 05 08:16:13.948294 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:16:13.948254 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:17:13.441718 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:13.441683 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:17:26.199150 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.199114 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj"] Sep 05 08:17:26.199738 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.199369 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" podUID="ee57f33b-65e7-459d-93b9-76292560fe93" containerName="maas-api" containerID="cri-o://65b3b56fae8dc8c366de22d51a6e6f223cb1138379286f1fc58ff5ee48b9e0bd" gracePeriod=30 Sep 05 08:17:26.231271 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.231232 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8c2e20-29809935-q7lk6"] Sep 05 08:17:26.233579 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.233553 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-8c2e20-29809935-q7lk6"] Sep 05 08:17:26.258779 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.258743 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22"] Sep 05 08:17:26.259123 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.259069 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" podUID="8872c321-9589-4d4e-b78f-1dcfa38b9462" containerName="payload-processing" containerID="cri-o://17e52ee8a5f4f632d954a14959f581960d4c272fb182d5eee654b36e05deb565" gracePeriod=30 Sep 05 08:17:26.267550 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.267519 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk"] Sep 05 08:17:26.267854 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.267817 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" podUID="bf0cb253-b79a-4906-8986-fcbdd291d8f9" containerName="payload-pre-processing" containerID="cri-o://55bc924ca617065291fb1328e7722ca121be543b87c9e8a1f9835b753e2938a0" gracePeriod=30 Sep 05 08:17:26.491399 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.491367 2576 generic.go:356] "Generic (PLEG): container finished" podID="bf0cb253-b79a-4906-8986-fcbdd291d8f9" containerID="55bc924ca617065291fb1328e7722ca121be543b87c9e8a1f9835b753e2938a0" exitCode=0 Sep 05 08:17:26.491609 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.491436 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" event={"ID":"bf0cb253-b79a-4906-8986-fcbdd291d8f9","Type":"ContainerDied","Data":"55bc924ca617065291fb1328e7722ca121be543b87c9e8a1f9835b753e2938a0"} Sep 05 08:17:26.493517 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.493485 2576 generic.go:356] "Generic (PLEG): container finished" podID="8872c321-9589-4d4e-b78f-1dcfa38b9462" containerID="17e52ee8a5f4f632d954a14959f581960d4c272fb182d5eee654b36e05deb565" exitCode=0 Sep 05 08:17:26.493665 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.493548 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" event={"ID":"8872c321-9589-4d4e-b78f-1dcfa38b9462","Type":"ContainerDied","Data":"17e52ee8a5f4f632d954a14959f581960d4c272fb182d5eee654b36e05deb565"} Sep 05 08:17:26.495577 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.495552 2576 generic.go:356] "Generic (PLEG): container finished" podID="ee57f33b-65e7-459d-93b9-76292560fe93" containerID="65b3b56fae8dc8c366de22d51a6e6f223cb1138379286f1fc58ff5ee48b9e0bd" exitCode=0 Sep 05 08:17:26.495707 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.495627 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" event={"ID":"ee57f33b-65e7-459d-93b9-76292560fe93","Type":"ContainerDied","Data":"65b3b56fae8dc8c366de22d51a6e6f223cb1138379286f1fc58ff5ee48b9e0bd"} Sep 05 08:17:26.529682 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.529656 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:17:26.562546 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.562521 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" Sep 05 08:17:26.569164 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.569133 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" Sep 05 08:17:26.657559 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.657521 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtgjn\" (UniqueName: \"kubernetes.io/projected/8872c321-9589-4d4e-b78f-1dcfa38b9462-kube-api-access-mtgjn\") pod \"8872c321-9589-4d4e-b78f-1dcfa38b9462\" (UID: \"8872c321-9589-4d4e-b78f-1dcfa38b9462\") " Sep 05 08:17:26.657559 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.657560 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpjpj\" (UniqueName: \"kubernetes.io/projected/bf0cb253-b79a-4906-8986-fcbdd291d8f9-kube-api-access-cpjpj\") pod \"bf0cb253-b79a-4906-8986-fcbdd291d8f9\" (UID: \"bf0cb253-b79a-4906-8986-fcbdd291d8f9\") " Sep 05 08:17:26.657826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.657580 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-maas-api-tls\") pod \"ee57f33b-65e7-459d-93b9-76292560fe93\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " Sep 05 08:17:26.657826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.657601 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-metrics-certs\") pod \"ee57f33b-65e7-459d-93b9-76292560fe93\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " Sep 05 08:17:26.657826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.657647 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn8m8\" (UniqueName: \"kubernetes.io/projected/ee57f33b-65e7-459d-93b9-76292560fe93-kube-api-access-zn8m8\") pod \"ee57f33b-65e7-459d-93b9-76292560fe93\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " Sep 05 08:17:26.657826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.657682 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee57f33b-65e7-459d-93b9-76292560fe93-openshift-service-ca\") pod \"ee57f33b-65e7-459d-93b9-76292560fe93\" (UID: \"ee57f33b-65e7-459d-93b9-76292560fe93\") " Sep 05 08:17:26.657826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.657708 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0cb253-b79a-4906-8986-fcbdd291d8f9-plugins-config-volume\") pod \"bf0cb253-b79a-4906-8986-fcbdd291d8f9\" (UID: \"bf0cb253-b79a-4906-8986-fcbdd291d8f9\") " Sep 05 08:17:26.657826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.657781 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8872c321-9589-4d4e-b78f-1dcfa38b9462-plugins-config-volume\") pod \"8872c321-9589-4d4e-b78f-1dcfa38b9462\" (UID: \"8872c321-9589-4d4e-b78f-1dcfa38b9462\") " Sep 05 08:17:26.658623 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.658309 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf0cb253-b79a-4906-8986-fcbdd291d8f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf0cb253-b79a-4906-8986-fcbdd291d8f9" (UID: "bf0cb253-b79a-4906-8986-fcbdd291d8f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:17:26.658623 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.658492 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8872c321-9589-4d4e-b78f-1dcfa38b9462-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8872c321-9589-4d4e-b78f-1dcfa38b9462" (UID: "8872c321-9589-4d4e-b78f-1dcfa38b9462"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:17:26.658804 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.658666 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee57f33b-65e7-459d-93b9-76292560fe93-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ee57f33b-65e7-459d-93b9-76292560fe93" (UID: "ee57f33b-65e7-459d-93b9-76292560fe93"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:17:26.660067 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.660044 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8872c321-9589-4d4e-b78f-1dcfa38b9462-kube-api-access-mtgjn" (OuterVolumeSpecName: "kube-api-access-mtgjn") pod "8872c321-9589-4d4e-b78f-1dcfa38b9462" (UID: "8872c321-9589-4d4e-b78f-1dcfa38b9462"). InnerVolumeSpecName "kube-api-access-mtgjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:17:26.660067 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.660051 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf0cb253-b79a-4906-8986-fcbdd291d8f9-kube-api-access-cpjpj" (OuterVolumeSpecName: "kube-api-access-cpjpj") pod "bf0cb253-b79a-4906-8986-fcbdd291d8f9" (UID: "bf0cb253-b79a-4906-8986-fcbdd291d8f9"). InnerVolumeSpecName "kube-api-access-cpjpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:17:26.660221 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.660063 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ee57f33b-65e7-459d-93b9-76292560fe93" (UID: "ee57f33b-65e7-459d-93b9-76292560fe93"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:17:26.660446 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.660428 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ee57f33b-65e7-459d-93b9-76292560fe93" (UID: "ee57f33b-65e7-459d-93b9-76292560fe93"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:17:26.660507 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.660440 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee57f33b-65e7-459d-93b9-76292560fe93-kube-api-access-zn8m8" (OuterVolumeSpecName: "kube-api-access-zn8m8") pod "ee57f33b-65e7-459d-93b9-76292560fe93" (UID: "ee57f33b-65e7-459d-93b9-76292560fe93"). InnerVolumeSpecName "kube-api-access-zn8m8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:17:26.759304 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.759171 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn8m8\" (UniqueName: \"kubernetes.io/projected/ee57f33b-65e7-459d-93b9-76292560fe93-kube-api-access-zn8m8\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:17:26.759304 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.759239 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee57f33b-65e7-459d-93b9-76292560fe93-openshift-service-ca\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:17:26.759304 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.759254 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf0cb253-b79a-4906-8986-fcbdd291d8f9-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:17:26.759304 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.759266 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8872c321-9589-4d4e-b78f-1dcfa38b9462-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:17:26.759304 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.759278 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtgjn\" (UniqueName: \"kubernetes.io/projected/8872c321-9589-4d4e-b78f-1dcfa38b9462-kube-api-access-mtgjn\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:17:26.759304 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.759289 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpjpj\" (UniqueName: \"kubernetes.io/projected/bf0cb253-b79a-4906-8986-fcbdd291d8f9-kube-api-access-cpjpj\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:17:26.759304 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.759301 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-maas-api-tls\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:17:26.759816 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:26.759312 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee57f33b-65e7-459d-93b9-76292560fe93-metrics-certs\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:17:27.502165 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.502073 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" Sep 05 08:17:27.502165 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.502089 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk" event={"ID":"bf0cb253-b79a-4906-8986-fcbdd291d8f9","Type":"ContainerDied","Data":"63581869568f411b94f14232150c1191cd2732c90a21f9fe135991f5f0233b82"} Sep 05 08:17:27.502165 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.502161 2576 scope.go:117] "RemoveContainer" containerID="55bc924ca617065291fb1328e7722ca121be543b87c9e8a1f9835b753e2938a0" Sep 05 08:17:27.503624 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.503576 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" Sep 05 08:17:27.503702 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.503573 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22" event={"ID":"8872c321-9589-4d4e-b78f-1dcfa38b9462","Type":"ContainerDied","Data":"2f99e4221a8031b3435ade933eb6995899604dc4d1508cc40447125b72763818"} Sep 05 08:17:27.505263 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.505236 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" event={"ID":"ee57f33b-65e7-459d-93b9-76292560fe93","Type":"ContainerDied","Data":"a8a3009ec743f82ab6238863827085b5ba1fd1f1c3294cbf6017a06a4f12e78e"} Sep 05 08:17:27.505419 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.505263 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj" Sep 05 08:17:27.513987 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.513967 2576 scope.go:117] "RemoveContainer" containerID="17e52ee8a5f4f632d954a14959f581960d4c272fb182d5eee654b36e05deb565" Sep 05 08:17:27.523001 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.522930 2576 scope.go:117] "RemoveContainer" containerID="65b3b56fae8dc8c366de22d51a6e6f223cb1138379286f1fc58ff5ee48b9e0bd" Sep 05 08:17:27.525415 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.525393 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk"] Sep 05 08:17:27.529811 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.529787 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-8c2e20-9f8c79846-djcpk"] Sep 05 08:17:27.539818 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.539792 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj"] Sep 05 08:17:27.544111 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.544090 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-8c2e20-cfbc57fcc-shqqj"] Sep 05 08:17:27.553727 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.553699 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22"] Sep 05 08:17:27.557498 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.557477 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-8c2e20-69b657c959-6cf22"] Sep 05 08:17:27.886308 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.886277 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de37b77-2b8c-47d2-b48a-582c40e25ac7" path="/var/lib/kubelet/pods/1de37b77-2b8c-47d2-b48a-582c40e25ac7/volumes" Sep 05 08:17:27.886609 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.886598 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8872c321-9589-4d4e-b78f-1dcfa38b9462" path="/var/lib/kubelet/pods/8872c321-9589-4d4e-b78f-1dcfa38b9462/volumes" Sep 05 08:17:27.886896 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.886885 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf0cb253-b79a-4906-8986-fcbdd291d8f9" path="/var/lib/kubelet/pods/bf0cb253-b79a-4906-8986-fcbdd291d8f9/volumes" Sep 05 08:17:27.887218 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:27.887197 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee57f33b-65e7-459d-93b9-76292560fe93" path="/var/lib/kubelet/pods/ee57f33b-65e7-459d-93b9-76292560fe93/volumes" Sep 05 08:17:29.747635 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:29.747587 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:17:33.348958 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.348920 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz"] Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349266 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee57f33b-65e7-459d-93b9-76292560fe93" containerName="maas-api" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349277 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee57f33b-65e7-459d-93b9-76292560fe93" containerName="maas-api" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349286 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09c92d84-96c9-4747-8d3e-03e5f937d0da" containerName="payload-pre-processing" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349293 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="09c92d84-96c9-4747-8d3e-03e5f937d0da" containerName="payload-pre-processing" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349306 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8872c321-9589-4d4e-b78f-1dcfa38b9462" containerName="payload-processing" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349311 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8872c321-9589-4d4e-b78f-1dcfa38b9462" containerName="payload-processing" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349318 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf0cb253-b79a-4906-8986-fcbdd291d8f9" containerName="payload-pre-processing" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349324 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf0cb253-b79a-4906-8986-fcbdd291d8f9" containerName="payload-pre-processing" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349330 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd11bb8d-6247-4503-8060-245e0dc7b769" containerName="payload-processing" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349335 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd11bb8d-6247-4503-8060-245e0dc7b769" containerName="payload-processing" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349341 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="361d498d-5224-4661-9767-81f305eedf6b" containerName="maas-api" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349347 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="361d498d-5224-4661-9767-81f305eedf6b" containerName="maas-api" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349358 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="117279a5-7719-430d-8e7e-2b39a9fe84c6" containerName="maas-api" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349365 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="117279a5-7719-430d-8e7e-2b39a9fe84c6" containerName="maas-api" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349375 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36d06474-f9f5-45bd-b520-dc182f585ef3" containerName="payload-processing" Sep 05 08:17:33.349388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349380 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="36d06474-f9f5-45bd-b520-dc182f585ef3" containerName="payload-processing" Sep 05 08:17:33.349878 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349448 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8872c321-9589-4d4e-b78f-1dcfa38b9462" containerName="payload-processing" Sep 05 08:17:33.349878 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349456 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="36d06474-f9f5-45bd-b520-dc182f585ef3" containerName="payload-processing" Sep 05 08:17:33.349878 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349462 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee57f33b-65e7-459d-93b9-76292560fe93" containerName="maas-api" Sep 05 08:17:33.349878 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349469 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf0cb253-b79a-4906-8986-fcbdd291d8f9" containerName="payload-pre-processing" Sep 05 08:17:33.349878 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349473 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd11bb8d-6247-4503-8060-245e0dc7b769" containerName="payload-processing" Sep 05 08:17:33.349878 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349480 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="09c92d84-96c9-4747-8d3e-03e5f937d0da" containerName="payload-pre-processing" Sep 05 08:17:33.349878 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349485 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="361d498d-5224-4661-9767-81f305eedf6b" containerName="maas-api" Sep 05 08:17:33.349878 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.349491 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="117279a5-7719-430d-8e7e-2b39a9fe84c6" containerName="maas-api" Sep 05 08:17:33.354609 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.354590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz" Sep 05 08:17:33.356818 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.356788 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mpmmp\"" Sep 05 08:17:33.357003 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.356825 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 08:17:33.358686 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.358664 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz"] Sep 05 08:17:33.521862 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.521821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842f36de-de8e-4df1-996f-47550809d147-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz\" (UID: \"842f36de-de8e-4df1-996f-47550809d147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz" Sep 05 08:17:33.623288 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.623185 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842f36de-de8e-4df1-996f-47550809d147-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz\" (UID: \"842f36de-de8e-4df1-996f-47550809d147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz" Sep 05 08:17:33.623846 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.623826 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842f36de-de8e-4df1-996f-47550809d147-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz\" (UID: \"842f36de-de8e-4df1-996f-47550809d147\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz" Sep 05 08:17:33.667269 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.667234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz" Sep 05 08:17:33.939197 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:33.939141 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz"] Sep 05 08:17:33.941590 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:17:33.941552 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod842f36de_de8e_4df1_996f_47550809d147.slice/crio-ab59fed184887f8525d040aae793a5a03414f1b44a1c616831465e24c1e4d00e WatchSource:0}: Error finding container ab59fed184887f8525d040aae793a5a03414f1b44a1c616831465e24c1e4d00e: Status 404 returned error can't find the container with id ab59fed184887f8525d040aae793a5a03414f1b44a1c616831465e24c1e4d00e Sep 05 08:17:34.549915 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:34.549813 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz" event={"ID":"842f36de-de8e-4df1-996f-47550809d147","Type":"ContainerStarted","Data":"d8a00c2ed4ff3f4c4f769d5863a1070ac64cfd0511c8ac37b089765ced09faac"} Sep 05 08:17:34.549915 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:34.549871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz" event={"ID":"842f36de-de8e-4df1-996f-47550809d147","Type":"ContainerStarted","Data":"ab59fed184887f8525d040aae793a5a03414f1b44a1c616831465e24c1e4d00e"} Sep 05 08:17:34.565361 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:34.565305 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz" podStartSLOduration=1.565289353 podStartE2EDuration="1.565289353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:17:34.563350059 +0000 UTC m=+1757.172325193" watchObservedRunningTime="2026-09-05 08:17:34.565289353 +0000 UTC m=+1757.174264551" Sep 05 08:17:35.556691 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:35.556660 2576 generic.go:356] "Generic (PLEG): container finished" podID="842f36de-de8e-4df1-996f-47550809d147" containerID="d8a00c2ed4ff3f4c4f769d5863a1070ac64cfd0511c8ac37b089765ced09faac" exitCode=0 Sep 05 08:17:35.557103 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:35.556744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz" event={"ID":"842f36de-de8e-4df1-996f-47550809d147","Type":"ContainerDied","Data":"d8a00c2ed4ff3f4c4f769d5863a1070ac64cfd0511c8ac37b089765ced09faac"} Sep 05 08:17:36.694330 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:36.694302 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz" Sep 05 08:17:36.853611 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:36.853576 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842f36de-de8e-4df1-996f-47550809d147-maas-api-service-ca\") pod \"842f36de-de8e-4df1-996f-47550809d147\" (UID: \"842f36de-de8e-4df1-996f-47550809d147\") " Sep 05 08:17:36.854003 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:36.853975 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/842f36de-de8e-4df1-996f-47550809d147-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "842f36de-de8e-4df1-996f-47550809d147" (UID: "842f36de-de8e-4df1-996f-47550809d147"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:17:36.954952 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:36.954912 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/842f36de-de8e-4df1-996f-47550809d147-maas-api-service-ca\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:17:37.570610 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:37.570577 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz" Sep 05 08:17:37.570794 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:37.570617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz" event={"ID":"842f36de-de8e-4df1-996f-47550809d147","Type":"ContainerDied","Data":"ab59fed184887f8525d040aae793a5a03414f1b44a1c616831465e24c1e4d00e"} Sep 05 08:17:37.570794 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:37.570670 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab59fed184887f8525d040aae793a5a03414f1b44a1c616831465e24c1e4d00e" Sep 05 08:17:43.777342 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:43.777305 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf"] Sep 05 08:17:43.777834 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:43.777611 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" podUID="78460e86-f41b-4f89-9d46-591534d4b527" containerName="payload-processing" containerID="cri-o://f02b97e9b3e3330798a43a89bcfe7e26c287cf56c41548ac0a94367e9e45e29e" gracePeriod=30 Sep 05 08:17:43.783314 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:43.783287 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp"] Sep 05 08:17:43.783562 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:43.783522 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" podUID="f4f7ea83-8b96-4c21-9b1b-4588968fcf6b" containerName="payload-pre-processing" containerID="cri-o://d7a46475fcdd27803e58f3591416061f69424903add993504b0782b4dfb9e698" gracePeriod=30 Sep 05 08:17:43.926621 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:43.926560 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz"] Sep 05 08:17:43.928685 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:43.928657 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-5ef325-kdznz"] Sep 05 08:17:44.065615 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.065585 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" Sep 05 08:17:44.068184 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.068165 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" Sep 05 08:17:44.105548 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.105440 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvkpk\" (UniqueName: \"kubernetes.io/projected/78460e86-f41b-4f89-9d46-591534d4b527-kube-api-access-jvkpk\") pod \"78460e86-f41b-4f89-9d46-591534d4b527\" (UID: \"78460e86-f41b-4f89-9d46-591534d4b527\") " Sep 05 08:17:44.106679 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.105563 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f7ea83-8b96-4c21-9b1b-4588968fcf6b-plugins-config-volume\") pod \"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b\" (UID: \"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b\") " Sep 05 08:17:44.106679 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.105640 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxc52\" (UniqueName: \"kubernetes.io/projected/f4f7ea83-8b96-4c21-9b1b-4588968fcf6b-kube-api-access-gxc52\") pod \"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b\" (UID: \"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b\") " Sep 05 08:17:44.106679 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.105668 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78460e86-f41b-4f89-9d46-591534d4b527-plugins-config-volume\") pod \"78460e86-f41b-4f89-9d46-591534d4b527\" (UID: \"78460e86-f41b-4f89-9d46-591534d4b527\") " Sep 05 08:17:44.106679 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.105991 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f7ea83-8b96-4c21-9b1b-4588968fcf6b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4f7ea83-8b96-4c21-9b1b-4588968fcf6b" (UID: "f4f7ea83-8b96-4c21-9b1b-4588968fcf6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:17:44.106679 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.106081 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4f7ea83-8b96-4c21-9b1b-4588968fcf6b-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:17:44.106679 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.106359 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78460e86-f41b-4f89-9d46-591534d4b527-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78460e86-f41b-4f89-9d46-591534d4b527" (UID: "78460e86-f41b-4f89-9d46-591534d4b527"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:17:44.108172 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.108150 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78460e86-f41b-4f89-9d46-591534d4b527-kube-api-access-jvkpk" (OuterVolumeSpecName: "kube-api-access-jvkpk") pod "78460e86-f41b-4f89-9d46-591534d4b527" (UID: "78460e86-f41b-4f89-9d46-591534d4b527"). InnerVolumeSpecName "kube-api-access-jvkpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:17:44.108172 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.108148 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f7ea83-8b96-4c21-9b1b-4588968fcf6b-kube-api-access-gxc52" (OuterVolumeSpecName: "kube-api-access-gxc52") pod "f4f7ea83-8b96-4c21-9b1b-4588968fcf6b" (UID: "f4f7ea83-8b96-4c21-9b1b-4588968fcf6b"). InnerVolumeSpecName "kube-api-access-gxc52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:17:44.206818 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.206780 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxc52\" (UniqueName: \"kubernetes.io/projected/f4f7ea83-8b96-4c21-9b1b-4588968fcf6b-kube-api-access-gxc52\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:17:44.206818 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.206815 2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78460e86-f41b-4f89-9d46-591534d4b527-plugins-config-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:17:44.207032 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.206832 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvkpk\" (UniqueName: \"kubernetes.io/projected/78460e86-f41b-4f89-9d46-591534d4b527-kube-api-access-jvkpk\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:17:44.610161 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.610125 2576 generic.go:356] "Generic (PLEG): container finished" podID="78460e86-f41b-4f89-9d46-591534d4b527" containerID="f02b97e9b3e3330798a43a89bcfe7e26c287cf56c41548ac0a94367e9e45e29e" exitCode=0 Sep 05 08:17:44.610454 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.610177 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" Sep 05 08:17:44.610454 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.610233 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" event={"ID":"78460e86-f41b-4f89-9d46-591534d4b527","Type":"ContainerDied","Data":"f02b97e9b3e3330798a43a89bcfe7e26c287cf56c41548ac0a94367e9e45e29e"} Sep 05 08:17:44.610454 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.610284 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf" event={"ID":"78460e86-f41b-4f89-9d46-591534d4b527","Type":"ContainerDied","Data":"8857cca39a0e9a64e25bc8fc00d5db4c0fde19f919a363669f41efe5b2e31928"} Sep 05 08:17:44.610454 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.610306 2576 scope.go:117] "RemoveContainer" containerID="f02b97e9b3e3330798a43a89bcfe7e26c287cf56c41548ac0a94367e9e45e29e" Sep 05 08:17:44.611767 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.611736 2576 generic.go:356] "Generic (PLEG): container finished" podID="f4f7ea83-8b96-4c21-9b1b-4588968fcf6b" containerID="d7a46475fcdd27803e58f3591416061f69424903add993504b0782b4dfb9e698" exitCode=0 Sep 05 08:17:44.612050 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.611776 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" Sep 05 08:17:44.612050 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.611853 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" event={"ID":"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b","Type":"ContainerDied","Data":"d7a46475fcdd27803e58f3591416061f69424903add993504b0782b4dfb9e698"} Sep 05 08:17:44.612050 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.611898 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp" event={"ID":"f4f7ea83-8b96-4c21-9b1b-4588968fcf6b","Type":"ContainerDied","Data":"1d119824926c34a50b114763ff675c04537a0dd167f30b6cc94148114e0faa6f"} Sep 05 08:17:44.620777 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.620759 2576 scope.go:117] "RemoveContainer" containerID="f02b97e9b3e3330798a43a89bcfe7e26c287cf56c41548ac0a94367e9e45e29e" Sep 05 08:17:44.621087 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:17:44.621065 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f02b97e9b3e3330798a43a89bcfe7e26c287cf56c41548ac0a94367e9e45e29e\": container with ID starting with f02b97e9b3e3330798a43a89bcfe7e26c287cf56c41548ac0a94367e9e45e29e not found: ID does not exist" containerID="f02b97e9b3e3330798a43a89bcfe7e26c287cf56c41548ac0a94367e9e45e29e" Sep 05 08:17:44.621175 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.621097 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f02b97e9b3e3330798a43a89bcfe7e26c287cf56c41548ac0a94367e9e45e29e"} err="failed to get container status \"f02b97e9b3e3330798a43a89bcfe7e26c287cf56c41548ac0a94367e9e45e29e\": rpc error: code = NotFound desc = could not find container \"f02b97e9b3e3330798a43a89bcfe7e26c287cf56c41548ac0a94367e9e45e29e\": container with ID starting with f02b97e9b3e3330798a43a89bcfe7e26c287cf56c41548ac0a94367e9e45e29e not found: ID does not exist" Sep 05 08:17:44.621175 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.621120 2576 scope.go:117] "RemoveContainer" containerID="d7a46475fcdd27803e58f3591416061f69424903add993504b0782b4dfb9e698" Sep 05 08:17:44.629554 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.629537 2576 scope.go:117] "RemoveContainer" containerID="d7a46475fcdd27803e58f3591416061f69424903add993504b0782b4dfb9e698" Sep 05 08:17:44.629838 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:17:44.629820 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7a46475fcdd27803e58f3591416061f69424903add993504b0782b4dfb9e698\": container with ID starting with d7a46475fcdd27803e58f3591416061f69424903add993504b0782b4dfb9e698 not found: ID does not exist" containerID="d7a46475fcdd27803e58f3591416061f69424903add993504b0782b4dfb9e698" Sep 05 08:17:44.629899 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.629844 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a46475fcdd27803e58f3591416061f69424903add993504b0782b4dfb9e698"} err="failed to get container status \"d7a46475fcdd27803e58f3591416061f69424903add993504b0782b4dfb9e698\": rpc error: code = NotFound desc = could not find container \"d7a46475fcdd27803e58f3591416061f69424903add993504b0782b4dfb9e698\": container with ID starting with d7a46475fcdd27803e58f3591416061f69424903add993504b0782b4dfb9e698 not found: ID does not exist" Sep 05 08:17:44.633799 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.633777 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp"] Sep 05 08:17:44.637740 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.637721 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-5ef325-5cc47d8bfd-c9nqp"] Sep 05 08:17:44.646456 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.646433 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf"] Sep 05 08:17:44.651971 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:44.651948 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5ef325-7dc8b66945-x2lcf"] Sep 05 08:17:45.885884 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:45.885852 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78460e86-f41b-4f89-9d46-591534d4b527" path="/var/lib/kubelet/pods/78460e86-f41b-4f89-9d46-591534d4b527/volumes" Sep 05 08:17:45.886310 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:45.886159 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="842f36de-de8e-4df1-996f-47550809d147" path="/var/lib/kubelet/pods/842f36de-de8e-4df1-996f-47550809d147/volumes" Sep 05 08:17:45.886492 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:45.886479 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4f7ea83-8b96-4c21-9b1b-4588968fcf6b" path="/var/lib/kubelet/pods/f4f7ea83-8b96-4c21-9b1b-4588968fcf6b/volumes" Sep 05 08:17:46.543591 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:46.543552 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:17:56.144388 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:17:56.144302 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:18:18.017095 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:18:18.017068 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 08:18:18.019203 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:18:18.019176 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 08:18:18.020535 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:18:18.020509 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 08:18:18.022617 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:18:18.022597 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 08:19:14.274753 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:19:14.274706 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:19:39.848726 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:19:39.848640 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:19:47.844460 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:19:47.844423 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:19:55.944277 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:19:55.944242 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:20:11.743308 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:11.743270 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:20:17.350081 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:17.350037 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc"] Sep 05 08:20:17.350555 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:17.350286 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" podUID="386f9de5-277f-4be8-bb35-4506e4451a6b" containerName="manager" containerID="cri-o://5e90f0655e104b23b9b0794d0cd1156f2c072d1a38abc7b2a8313193ac5bac77" gracePeriod=10 Sep 05 08:20:20.096595 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.096570 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" Sep 05 08:20:20.160826 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.160791 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znhjc\" (UniqueName: \"kubernetes.io/projected/386f9de5-277f-4be8-bb35-4506e4451a6b-kube-api-access-znhjc\") pod \"386f9de5-277f-4be8-bb35-4506e4451a6b\" (UID: \"386f9de5-277f-4be8-bb35-4506e4451a6b\") " Sep 05 08:20:20.160998 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.160904 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/386f9de5-277f-4be8-bb35-4506e4451a6b-extensions-socket-volume\") pod \"386f9de5-277f-4be8-bb35-4506e4451a6b\" (UID: \"386f9de5-277f-4be8-bb35-4506e4451a6b\") " Sep 05 08:20:20.161307 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.161286 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/386f9de5-277f-4be8-bb35-4506e4451a6b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "386f9de5-277f-4be8-bb35-4506e4451a6b" (UID: "386f9de5-277f-4be8-bb35-4506e4451a6b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 08:20:20.163082 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.163054 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386f9de5-277f-4be8-bb35-4506e4451a6b-kube-api-access-znhjc" (OuterVolumeSpecName: "kube-api-access-znhjc") pod "386f9de5-277f-4be8-bb35-4506e4451a6b" (UID: "386f9de5-277f-4be8-bb35-4506e4451a6b"). InnerVolumeSpecName "kube-api-access-znhjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:20:20.261778 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.261686 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/386f9de5-277f-4be8-bb35-4506e4451a6b-extensions-socket-volume\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:20:20.261778 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.261720 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znhjc\" (UniqueName: \"kubernetes.io/projected/386f9de5-277f-4be8-bb35-4506e4451a6b-kube-api-access-znhjc\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:20:20.435156 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.435117 2576 generic.go:356] "Generic (PLEG): container finished" podID="386f9de5-277f-4be8-bb35-4506e4451a6b" containerID="5e90f0655e104b23b9b0794d0cd1156f2c072d1a38abc7b2a8313193ac5bac77" exitCode=0 Sep 05 08:20:20.435367 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.435167 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" Sep 05 08:20:20.435367 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.435168 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" event={"ID":"386f9de5-277f-4be8-bb35-4506e4451a6b","Type":"ContainerDied","Data":"5e90f0655e104b23b9b0794d0cd1156f2c072d1a38abc7b2a8313193ac5bac77"} Sep 05 08:20:20.435367 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.435258 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" event={"ID":"386f9de5-277f-4be8-bb35-4506e4451a6b","Type":"ContainerDied","Data":"7262a46343318ea74dfc59acf3f9e5cfb71ea43c66b16646d9fc1824908b611b"} Sep 05 08:20:20.435367 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.435289 2576 scope.go:117] "RemoveContainer" containerID="5e90f0655e104b23b9b0794d0cd1156f2c072d1a38abc7b2a8313193ac5bac77" Sep 05 08:20:20.445472 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.445452 2576 scope.go:117] "RemoveContainer" containerID="5e90f0655e104b23b9b0794d0cd1156f2c072d1a38abc7b2a8313193ac5bac77" Sep 05 08:20:20.445744 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:20:20.445723 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e90f0655e104b23b9b0794d0cd1156f2c072d1a38abc7b2a8313193ac5bac77\": container with ID starting with 5e90f0655e104b23b9b0794d0cd1156f2c072d1a38abc7b2a8313193ac5bac77 not found: ID does not exist" containerID="5e90f0655e104b23b9b0794d0cd1156f2c072d1a38abc7b2a8313193ac5bac77" Sep 05 08:20:20.445792 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.445753 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e90f0655e104b23b9b0794d0cd1156f2c072d1a38abc7b2a8313193ac5bac77"} err="failed to get container status \"5e90f0655e104b23b9b0794d0cd1156f2c072d1a38abc7b2a8313193ac5bac77\": rpc error: code = NotFound desc = could not find container \"5e90f0655e104b23b9b0794d0cd1156f2c072d1a38abc7b2a8313193ac5bac77\": container with ID starting with 5e90f0655e104b23b9b0794d0cd1156f2c072d1a38abc7b2a8313193ac5bac77 not found: ID does not exist" Sep 05 08:20:20.468775 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.468742 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc"] Sep 05 08:20:20.470707 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:20.470688 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc"] Sep 05 08:20:21.049338 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:21.049297 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-28dlc" podUID="386f9de5-277f-4be8-bb35-4506e4451a6b" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.35:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 05 08:20:21.886409 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:20:21.886376 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="386f9de5-277f-4be8-bb35-4506e4451a6b" path="/var/lib/kubelet/pods/386f9de5-277f-4be8-bb35-4506e4451a6b/volumes" Sep 05 08:21:18.066371 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.066291 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6"] Sep 05 08:21:18.066786 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.066622 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="842f36de-de8e-4df1-996f-47550809d147" containerName="revoke-keys" Sep 05 08:21:18.066786 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.066635 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="842f36de-de8e-4df1-996f-47550809d147" containerName="revoke-keys" Sep 05 08:21:18.066786 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.066654 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="386f9de5-277f-4be8-bb35-4506e4451a6b" containerName="manager" Sep 05 08:21:18.066786 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.066661 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="386f9de5-277f-4be8-bb35-4506e4451a6b" containerName="manager" Sep 05 08:21:18.066786 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.066675 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4f7ea83-8b96-4c21-9b1b-4588968fcf6b" containerName="payload-pre-processing" Sep 05 08:21:18.066786 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.066681 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4f7ea83-8b96-4c21-9b1b-4588968fcf6b" containerName="payload-pre-processing" Sep 05 08:21:18.066786 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.066688 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78460e86-f41b-4f89-9d46-591534d4b527" containerName="payload-processing" Sep 05 08:21:18.066786 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.066693 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="78460e86-f41b-4f89-9d46-591534d4b527" containerName="payload-processing" Sep 05 08:21:18.066786 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.066759 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4f7ea83-8b96-4c21-9b1b-4588968fcf6b" containerName="payload-pre-processing" Sep 05 08:21:18.066786 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.066766 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="842f36de-de8e-4df1-996f-47550809d147" containerName="revoke-keys" Sep 05 08:21:18.066786 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.066772 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="386f9de5-277f-4be8-bb35-4506e4451a6b" containerName="manager" Sep 05 08:21:18.066786 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.066780 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="78460e86-f41b-4f89-9d46-591534d4b527" containerName="payload-processing" Sep 05 08:21:18.069617 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.069601 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6" Sep 05 08:21:18.071874 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.071855 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dr9bw\"" Sep 05 08:21:18.085773 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.085748 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6"] Sep 05 08:21:18.151731 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.151700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk4dh\" (UniqueName: \"kubernetes.io/projected/b73aad89-bbde-488e-8d5d-5866a5d6f4c8-kube-api-access-dk4dh\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-ps5q6\" (UID: \"b73aad89-bbde-488e-8d5d-5866a5d6f4c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6" Sep 05 08:21:18.151906 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.151740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b73aad89-bbde-488e-8d5d-5866a5d6f4c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-ps5q6\" (UID: \"b73aad89-bbde-488e-8d5d-5866a5d6f4c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6" Sep 05 08:21:18.238380 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.238346 2576 scope.go:117] "RemoveContainer" containerID="15a9e459e384772a7be5235b74b145836be4aa48e60b157e30c7e1aa66c9da67" Sep 05 08:21:18.247450 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.247431 2576 scope.go:117] "RemoveContainer" containerID="799c1949036b2dc5893c1421190ddb06da2729b52256547df4b53b909b6b3b9d" Sep 05 08:21:18.252700 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.252671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk4dh\" (UniqueName: \"kubernetes.io/projected/b73aad89-bbde-488e-8d5d-5866a5d6f4c8-kube-api-access-dk4dh\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-ps5q6\" (UID: \"b73aad89-bbde-488e-8d5d-5866a5d6f4c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6" Sep 05 08:21:18.252812 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.252704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b73aad89-bbde-488e-8d5d-5866a5d6f4c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-ps5q6\" (UID: \"b73aad89-bbde-488e-8d5d-5866a5d6f4c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6" Sep 05 08:21:18.253069 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.253047 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b73aad89-bbde-488e-8d5d-5866a5d6f4c8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-ps5q6\" (UID: \"b73aad89-bbde-488e-8d5d-5866a5d6f4c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6" Sep 05 08:21:18.261132 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.261112 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk4dh\" (UniqueName: \"kubernetes.io/projected/b73aad89-bbde-488e-8d5d-5866a5d6f4c8-kube-api-access-dk4dh\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-ps5q6\" (UID: \"b73aad89-bbde-488e-8d5d-5866a5d6f4c8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6" Sep 05 08:21:18.379679 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.379563 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6" Sep 05 08:21:18.667706 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.667682 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6"] Sep 05 08:21:18.669127 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:21:18.669098 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb73aad89_bbde_488e_8d5d_5866a5d6f4c8.slice/crio-81cc58c79833e1561474679e90611e33d2b0cbf8b2a125f8fbc981a7a1e9be19 WatchSource:0}: Error finding container 81cc58c79833e1561474679e90611e33d2b0cbf8b2a125f8fbc981a7a1e9be19: Status 404 returned error can't find the container with id 81cc58c79833e1561474679e90611e33d2b0cbf8b2a125f8fbc981a7a1e9be19 Sep 05 08:21:18.748917 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.748882 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6" event={"ID":"b73aad89-bbde-488e-8d5d-5866a5d6f4c8","Type":"ContainerStarted","Data":"4a6cf4f0d545618d497befbf6dde92ea3eb532b0a22ddcca5d8570ea246b3676"} Sep 05 08:21:18.749081 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.748973 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6" event={"ID":"b73aad89-bbde-488e-8d5d-5866a5d6f4c8","Type":"ContainerStarted","Data":"81cc58c79833e1561474679e90611e33d2b0cbf8b2a125f8fbc981a7a1e9be19"} Sep 05 08:21:18.749081 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.749005 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6" Sep 05 08:21:18.767723 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:18.767679 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6" podStartSLOduration=0.767663157 podStartE2EDuration="767.663157ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:21:18.766032511 +0000 UTC m=+1981.375007633" watchObservedRunningTime="2026-09-05 08:21:18.767663157 +0000 UTC m=+1981.376638279" Sep 05 08:21:29.756671 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:29.756641 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-ps5q6" Sep 05 08:21:37.743297 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:37.743248 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:21:46.144972 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:46.144932 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:21:54.042891 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:21:54.042856 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:22:10.482674 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.482640 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz"] Sep 05 08:22:10.486140 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.486118 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.488119 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.488100 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-cc5f87b2-kserve-self-signed-certs\"" Sep 05 08:22:10.495327 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.495307 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz"] Sep 05 08:22:10.574176 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.574145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-tls-certs\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.574324 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.574180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkrn5\" (UniqueName: \"kubernetes.io/projected/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-kube-api-access-hkrn5\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.574324 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.574244 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-dshm\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.574324 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.574314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-home\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.574459 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.574332 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-model-cache\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.574459 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.574348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-tmp-dir\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.675599 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.675564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-home\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.675747 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.675649 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-model-cache\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.675747 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.675705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-tmp-dir\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.675866 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.675782 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-tls-certs\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.675866 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.675806 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkrn5\" (UniqueName: \"kubernetes.io/projected/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-kube-api-access-hkrn5\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.675866 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.675840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-dshm\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.676034 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.675925 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-home\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.676034 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.675984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-model-cache\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.676124 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.676046 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-tmp-dir\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.678150 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.678129 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-dshm\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.678470 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.678450 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-tls-certs\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.683634 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.683615 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkrn5\" (UniqueName: \"kubernetes.io/projected/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-kube-api-access-hkrn5\") pod \"e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:10.796125 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:10.796039 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:11.072247 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:11.067182 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz"] Sep 05 08:22:11.073941 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:22:11.073912 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc9e529c_abbc_4d56_9fa5_dd7ed3d770fb.slice/crio-432318e02f21258c46c9f417a87bd27dea2da69681eebf7066d972c5e1b64925 WatchSource:0}: Error finding container 432318e02f21258c46c9f417a87bd27dea2da69681eebf7066d972c5e1b64925: Status 404 returned error can't find the container with id 432318e02f21258c46c9f417a87bd27dea2da69681eebf7066d972c5e1b64925 Sep 05 08:22:12.009902 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:12.009867 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" event={"ID":"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb","Type":"ContainerStarted","Data":"1f24281a516a029399826c1bf290c5a1d5c03f271525d8ec5e700dd6aaac7d25"} Sep 05 08:22:12.009902 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:12.009905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" event={"ID":"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb","Type":"ContainerStarted","Data":"432318e02f21258c46c9f417a87bd27dea2da69681eebf7066d972c5e1b64925"} Sep 05 08:22:12.026488 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:12.026416 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" podStartSLOduration=2.026399974 podStartE2EDuration="2.026399974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:22:12.024299249 +0000 UTC m=+2034.633274371" watchObservedRunningTime="2026-09-05 08:22:12.026399974 +0000 UTC m=+2034.635375096" Sep 05 08:22:12.844164 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:12.844123 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:22:20.796562 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:20.796517 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:20.797055 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:20.796576 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:20.809194 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:20.809167 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:21.066905 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:21.066884 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:23.844726 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:23.844686 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:22:40.570990 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:40.570904 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz"] Sep 05 08:22:42.639370 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:42.639332 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:22:51.640158 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:51.640113 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:22:55.592010 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.591952 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" podUID="fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb" containerName="main" containerID="cri-o://1f24281a516a029399826c1bf290c5a1d5c03f271525d8ec5e700dd6aaac7d25" gracePeriod=45 Sep 05 08:22:55.840540 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.840513 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:55.958809 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.958701 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-tmp-dir\") pod \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " Sep 05 08:22:55.958809 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.958784 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-home\") pod \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " Sep 05 08:22:55.959076 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.958822 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-tls-certs\") pod \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " Sep 05 08:22:55.959076 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.958851 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-dshm\") pod \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " Sep 05 08:22:55.959076 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.958985 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkrn5\" (UniqueName: \"kubernetes.io/projected/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-kube-api-access-hkrn5\") pod \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " Sep 05 08:22:55.959076 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.959005 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb" (UID: "fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 08:22:55.959076 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.959039 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-model-cache\") pod \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\" (UID: \"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb\") " Sep 05 08:22:55.959340 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.959098 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-home" (OuterVolumeSpecName: "home") pod "fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb" (UID: "fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 08:22:55.959340 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.959191 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb" (UID: "fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 08:22:55.959340 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.959311 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-model-cache\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:22:55.959340 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.959324 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-tmp-dir\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:22:55.959340 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.959335 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-home\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:22:55.961319 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.961289 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-dshm" (OuterVolumeSpecName: "dshm") pod "fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb" (UID: "fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 05 08:22:55.961409 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.961359 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb" (UID: "fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:22:55.961409 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:55.961364 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-kube-api-access-hkrn5" (OuterVolumeSpecName: "kube-api-access-hkrn5") pod "fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb" (UID: "fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb"). InnerVolumeSpecName "kube-api-access-hkrn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:22:56.060124 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:56.060085 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-tls-certs\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:22:56.060124 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:56.060115 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-dshm\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:22:56.060124 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:56.060124 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkrn5\" (UniqueName: \"kubernetes.io/projected/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb-kube-api-access-hkrn5\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:22:56.240474 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:56.240380 2576 generic.go:356] "Generic (PLEG): container finished" podID="fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb" containerID="1f24281a516a029399826c1bf290c5a1d5c03f271525d8ec5e700dd6aaac7d25" exitCode=0 Sep 05 08:22:56.240474 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:56.240420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" event={"ID":"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb","Type":"ContainerDied","Data":"1f24281a516a029399826c1bf290c5a1d5c03f271525d8ec5e700dd6aaac7d25"} Sep 05 08:22:56.240474 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:56.240425 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" Sep 05 08:22:56.240474 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:56.240449 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz" event={"ID":"fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb","Type":"ContainerDied","Data":"432318e02f21258c46c9f417a87bd27dea2da69681eebf7066d972c5e1b64925"} Sep 05 08:22:56.240474 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:56.240468 2576 scope.go:117] "RemoveContainer" containerID="1f24281a516a029399826c1bf290c5a1d5c03f271525d8ec5e700dd6aaac7d25" Sep 05 08:22:56.250174 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:56.250157 2576 scope.go:117] "RemoveContainer" containerID="1f24281a516a029399826c1bf290c5a1d5c03f271525d8ec5e700dd6aaac7d25" Sep 05 08:22:56.250533 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:22:56.250505 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f24281a516a029399826c1bf290c5a1d5c03f271525d8ec5e700dd6aaac7d25\": container with ID starting with 1f24281a516a029399826c1bf290c5a1d5c03f271525d8ec5e700dd6aaac7d25 not found: ID does not exist" containerID="1f24281a516a029399826c1bf290c5a1d5c03f271525d8ec5e700dd6aaac7d25" Sep 05 08:22:56.250661 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:56.250544 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f24281a516a029399826c1bf290c5a1d5c03f271525d8ec5e700dd6aaac7d25"} err="failed to get container status \"1f24281a516a029399826c1bf290c5a1d5c03f271525d8ec5e700dd6aaac7d25\": rpc error: code = NotFound desc = could not find container \"1f24281a516a029399826c1bf290c5a1d5c03f271525d8ec5e700dd6aaac7d25\": container with ID starting with 1f24281a516a029399826c1bf290c5a1d5c03f271525d8ec5e700dd6aaac7d25 not found: ID does not exist" Sep 05 08:22:56.259550 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:56.259517 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz"] Sep 05 08:22:56.265282 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:56.265255 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-cc5f87b2-kserve-6ff478ffb7-q2xsz"] Sep 05 08:22:57.886338 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:57.886302 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb" path="/var/lib/kubelet/pods/fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb/volumes" Sep 05 08:22:58.642657 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:22:58.642621 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:23:05.512233 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.512187 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw"] Sep 05 08:23:05.512684 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.512549 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb" containerName="main" Sep 05 08:23:05.512684 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.512564 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb" containerName="main" Sep 05 08:23:05.512684 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.512642 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc9e529c-abbc-4d56-9fa5-dd7ed3d770fb" containerName="main" Sep 05 08:23:05.517092 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.517067 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.520289 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.520260 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 05 08:23:05.520437 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.520288 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 05 08:23:05.520514 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.520476 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-c62fz\"" Sep 05 08:23:05.520514 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.520481 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 05 08:23:05.520617 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.520587 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 05 08:23:05.524456 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.524434 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw"] Sep 05 08:23:05.528624 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.528600 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g"] Sep 05 08:23:05.531941 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.531920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:05.540720 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.540696 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g"] Sep 05 08:23:05.641584 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.641544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc73b286-ca2a-42b3-bc32-21ba59b095a6-metrics-certs\") pod \"maas-api-7895c5cf96-9zr5g\" (UID: \"bc73b286-ca2a-42b3-bc32-21ba59b095a6\") " pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:05.641783 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.641599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc73b286-ca2a-42b3-bc32-21ba59b095a6-maas-api-tls\") pod \"maas-api-7895c5cf96-9zr5g\" (UID: \"bc73b286-ca2a-42b3-bc32-21ba59b095a6\") " pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:05.641783 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.641634 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc73b286-ca2a-42b3-bc32-21ba59b095a6-openshift-service-ca\") pod \"maas-api-7895c5cf96-9zr5g\" (UID: \"bc73b286-ca2a-42b3-bc32-21ba59b095a6\") " pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:05.641783 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.641671 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd1dd31-3567-42ce-9f58-282db972f7b0-maas-api-tls\") pod \"maas-api-7548c976dd-f7ggw\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.641783 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.641685 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd1dd31-3567-42ce-9f58-282db972f7b0-openshift-service-ca\") pod \"maas-api-7548c976dd-f7ggw\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.641783 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.641737 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnc7j\" (UniqueName: \"kubernetes.io/projected/bc73b286-ca2a-42b3-bc32-21ba59b095a6-kube-api-access-nnc7j\") pod \"maas-api-7895c5cf96-9zr5g\" (UID: \"bc73b286-ca2a-42b3-bc32-21ba59b095a6\") " pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:05.641783 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.641776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cmk8\" (UniqueName: \"kubernetes.io/projected/9cd1dd31-3567-42ce-9f58-282db972f7b0-kube-api-access-9cmk8\") pod \"maas-api-7548c976dd-f7ggw\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.642012 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.641805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd1dd31-3567-42ce-9f58-282db972f7b0-metrics-certs\") pod \"maas-api-7548c976dd-f7ggw\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.739552 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.739509 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:23:05.742163 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.742140 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnc7j\" (UniqueName: \"kubernetes.io/projected/bc73b286-ca2a-42b3-bc32-21ba59b095a6-kube-api-access-nnc7j\") pod \"maas-api-7895c5cf96-9zr5g\" (UID: \"bc73b286-ca2a-42b3-bc32-21ba59b095a6\") " pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:05.742258 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.742174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cmk8\" (UniqueName: \"kubernetes.io/projected/9cd1dd31-3567-42ce-9f58-282db972f7b0-kube-api-access-9cmk8\") pod \"maas-api-7548c976dd-f7ggw\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.742258 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.742195 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd1dd31-3567-42ce-9f58-282db972f7b0-metrics-certs\") pod \"maas-api-7548c976dd-f7ggw\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.742369 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.742261 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc73b286-ca2a-42b3-bc32-21ba59b095a6-metrics-certs\") pod \"maas-api-7895c5cf96-9zr5g\" (UID: \"bc73b286-ca2a-42b3-bc32-21ba59b095a6\") " pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:05.742369 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.742308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc73b286-ca2a-42b3-bc32-21ba59b095a6-maas-api-tls\") pod \"maas-api-7895c5cf96-9zr5g\" (UID: \"bc73b286-ca2a-42b3-bc32-21ba59b095a6\") " pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:05.742369 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.742344 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc73b286-ca2a-42b3-bc32-21ba59b095a6-openshift-service-ca\") pod \"maas-api-7895c5cf96-9zr5g\" (UID: \"bc73b286-ca2a-42b3-bc32-21ba59b095a6\") " pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:05.742522 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.742373 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd1dd31-3567-42ce-9f58-282db972f7b0-maas-api-tls\") pod \"maas-api-7548c976dd-f7ggw\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.742522 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.742395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd1dd31-3567-42ce-9f58-282db972f7b0-openshift-service-ca\") pod \"maas-api-7548c976dd-f7ggw\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.743404 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.743375 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd1dd31-3567-42ce-9f58-282db972f7b0-openshift-service-ca\") pod \"maas-api-7548c976dd-f7ggw\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.743547 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.743436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc73b286-ca2a-42b3-bc32-21ba59b095a6-openshift-service-ca\") pod \"maas-api-7895c5cf96-9zr5g\" (UID: \"bc73b286-ca2a-42b3-bc32-21ba59b095a6\") " pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:05.745154 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.745085 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc73b286-ca2a-42b3-bc32-21ba59b095a6-metrics-certs\") pod \"maas-api-7895c5cf96-9zr5g\" (UID: \"bc73b286-ca2a-42b3-bc32-21ba59b095a6\") " pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:05.745154 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.745095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd1dd31-3567-42ce-9f58-282db972f7b0-metrics-certs\") pod \"maas-api-7548c976dd-f7ggw\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.745154 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.745091 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd1dd31-3567-42ce-9f58-282db972f7b0-maas-api-tls\") pod \"maas-api-7548c976dd-f7ggw\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.745356 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.745174 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc73b286-ca2a-42b3-bc32-21ba59b095a6-maas-api-tls\") pod \"maas-api-7895c5cf96-9zr5g\" (UID: \"bc73b286-ca2a-42b3-bc32-21ba59b095a6\") " pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:05.749196 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.749170 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnc7j\" (UniqueName: \"kubernetes.io/projected/bc73b286-ca2a-42b3-bc32-21ba59b095a6-kube-api-access-nnc7j\") pod \"maas-api-7895c5cf96-9zr5g\" (UID: \"bc73b286-ca2a-42b3-bc32-21ba59b095a6\") " pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:05.749338 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.749320 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cmk8\" (UniqueName: \"kubernetes.io/projected/9cd1dd31-3567-42ce-9f58-282db972f7b0-kube-api-access-9cmk8\") pod \"maas-api-7548c976dd-f7ggw\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.829188 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.829156 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:05.843392 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:05.843364 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:06.109995 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:06.109956 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw"] Sep 05 08:23:06.112603 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:23:06.112575 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd1dd31_3567_42ce_9f58_282db972f7b0.slice/crio-9bf1a61e8e34b4f45eab8c3756a7bbc9f5658decfc3c03b046f37b36349f83d7 WatchSource:0}: Error finding container 9bf1a61e8e34b4f45eab8c3756a7bbc9f5658decfc3c03b046f37b36349f83d7: Status 404 returned error can't find the container with id 9bf1a61e8e34b4f45eab8c3756a7bbc9f5658decfc3c03b046f37b36349f83d7 Sep 05 08:23:06.113968 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:06.113950 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 05 08:23:06.130465 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:06.130443 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g"] Sep 05 08:23:06.132323 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:23:06.132295 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc73b286_ca2a_42b3_bc32_21ba59b095a6.slice/crio-aa5066f04f02861b6f1810c3ada1561d25b41f3d490be65ebd1748a40ac3dbc9 WatchSource:0}: Error finding container aa5066f04f02861b6f1810c3ada1561d25b41f3d490be65ebd1748a40ac3dbc9: Status 404 returned error can't find the container with id aa5066f04f02861b6f1810c3ada1561d25b41f3d490be65ebd1748a40ac3dbc9 Sep 05 08:23:06.296304 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:06.296267 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" event={"ID":"9cd1dd31-3567-42ce-9f58-282db972f7b0","Type":"ContainerStarted","Data":"9bf1a61e8e34b4f45eab8c3756a7bbc9f5658decfc3c03b046f37b36349f83d7"} Sep 05 08:23:06.297537 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:06.297511 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" event={"ID":"bc73b286-ca2a-42b3-bc32-21ba59b095a6","Type":"ContainerStarted","Data":"aa5066f04f02861b6f1810c3ada1561d25b41f3d490be65ebd1748a40ac3dbc9"} Sep 05 08:23:07.303901 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:07.303858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" event={"ID":"9cd1dd31-3567-42ce-9f58-282db972f7b0","Type":"ContainerStarted","Data":"ec90c05704989f88f4426dd5f25430ddaff75b8e4e1f05fe1652f1f3de28ead5"} Sep 05 08:23:07.304397 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:07.303992 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:07.305474 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:07.305451 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" event={"ID":"bc73b286-ca2a-42b3-bc32-21ba59b095a6","Type":"ContainerStarted","Data":"7ab6fc34e8ecca8c302d74e62ab6752cfd159e256800b7975b2cc0c869d48fdc"} Sep 05 08:23:07.305596 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:07.305534 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:07.320200 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:07.320154 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" podStartSLOduration=1.812323203 podStartE2EDuration="2.320139774s" totalImagesPullingTime="507.816571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:23:07.318275108 +0000 UTC m=+2089.927250232" watchObservedRunningTime="2026-09-05 08:23:07.320139774 +0000 UTC m=+2089.929114897" Sep 05 08:23:07.332360 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:07.332307 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" podStartSLOduration=1.831672929 podStartE2EDuration="2.332289247s" totalImagesPullingTime="500.616318ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:23:07.332060232 +0000 UTC m=+2089.941035356" watchObservedRunningTime="2026-09-05 08:23:07.332289247 +0000 UTC m=+2089.941264371" Sep 05 08:23:13.144143 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.144105 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:23:13.319490 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.319465 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7895c5cf96-9zr5g" Sep 05 08:23:13.319788 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.319770 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:13.379428 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.379394 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw"] Sep 05 08:23:13.379646 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.379586 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" podUID="9cd1dd31-3567-42ce-9f58-282db972f7b0" containerName="maas-api" containerID="cri-o://ec90c05704989f88f4426dd5f25430ddaff75b8e4e1f05fe1652f1f3de28ead5" gracePeriod=30 Sep 05 08:23:13.626765 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.626733 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:13.813858 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.813828 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd1dd31-3567-42ce-9f58-282db972f7b0-metrics-certs\") pod \"9cd1dd31-3567-42ce-9f58-282db972f7b0\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " Sep 05 08:23:13.814006 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.813884 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd1dd31-3567-42ce-9f58-282db972f7b0-maas-api-tls\") pod \"9cd1dd31-3567-42ce-9f58-282db972f7b0\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " Sep 05 08:23:13.814006 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.813938 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cmk8\" (UniqueName: \"kubernetes.io/projected/9cd1dd31-3567-42ce-9f58-282db972f7b0-kube-api-access-9cmk8\") pod \"9cd1dd31-3567-42ce-9f58-282db972f7b0\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " Sep 05 08:23:13.814006 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.813994 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd1dd31-3567-42ce-9f58-282db972f7b0-openshift-service-ca\") pod \"9cd1dd31-3567-42ce-9f58-282db972f7b0\" (UID: \"9cd1dd31-3567-42ce-9f58-282db972f7b0\") " Sep 05 08:23:13.814475 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.814447 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd1dd31-3567-42ce-9f58-282db972f7b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9cd1dd31-3567-42ce-9f58-282db972f7b0" (UID: "9cd1dd31-3567-42ce-9f58-282db972f7b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 05 08:23:13.816235 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.816197 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd1dd31-3567-42ce-9f58-282db972f7b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9cd1dd31-3567-42ce-9f58-282db972f7b0" (UID: "9cd1dd31-3567-42ce-9f58-282db972f7b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:23:13.816328 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.816258 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd1dd31-3567-42ce-9f58-282db972f7b0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9cd1dd31-3567-42ce-9f58-282db972f7b0" (UID: "9cd1dd31-3567-42ce-9f58-282db972f7b0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 05 08:23:13.816328 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.816308 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd1dd31-3567-42ce-9f58-282db972f7b0-kube-api-access-9cmk8" (OuterVolumeSpecName: "kube-api-access-9cmk8") pod "9cd1dd31-3567-42ce-9f58-282db972f7b0" (UID: "9cd1dd31-3567-42ce-9f58-282db972f7b0"). InnerVolumeSpecName "kube-api-access-9cmk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 05 08:23:13.915377 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.915346 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cd1dd31-3567-42ce-9f58-282db972f7b0-metrics-certs\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:23:13.915377 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.915372 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9cd1dd31-3567-42ce-9f58-282db972f7b0-maas-api-tls\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:23:13.915583 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.915401 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cmk8\" (UniqueName: \"kubernetes.io/projected/9cd1dd31-3567-42ce-9f58-282db972f7b0-kube-api-access-9cmk8\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:23:13.915583 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:13.915412 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd1dd31-3567-42ce-9f58-282db972f7b0-openshift-service-ca\") on node \"ip-10-0-143-184.ec2.internal\" DevicePath \"\"" Sep 05 08:23:14.344710 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:14.344675 2576 generic.go:356] "Generic (PLEG): container finished" podID="9cd1dd31-3567-42ce-9f58-282db972f7b0" containerID="ec90c05704989f88f4426dd5f25430ddaff75b8e4e1f05fe1652f1f3de28ead5" exitCode=0 Sep 05 08:23:14.345129 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:14.344744 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" Sep 05 08:23:14.345129 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:14.344769 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" event={"ID":"9cd1dd31-3567-42ce-9f58-282db972f7b0","Type":"ContainerDied","Data":"ec90c05704989f88f4426dd5f25430ddaff75b8e4e1f05fe1652f1f3de28ead5"} Sep 05 08:23:14.345129 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:14.344859 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw" event={"ID":"9cd1dd31-3567-42ce-9f58-282db972f7b0","Type":"ContainerDied","Data":"9bf1a61e8e34b4f45eab8c3756a7bbc9f5658decfc3c03b046f37b36349f83d7"} Sep 05 08:23:14.345129 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:14.344888 2576 scope.go:117] "RemoveContainer" containerID="ec90c05704989f88f4426dd5f25430ddaff75b8e4e1f05fe1652f1f3de28ead5" Sep 05 08:23:14.353970 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:14.353953 2576 scope.go:117] "RemoveContainer" containerID="ec90c05704989f88f4426dd5f25430ddaff75b8e4e1f05fe1652f1f3de28ead5" Sep 05 08:23:14.354242 ip-10-0-143-184 kubenswrapper[2576]: E0905 08:23:14.354205 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec90c05704989f88f4426dd5f25430ddaff75b8e4e1f05fe1652f1f3de28ead5\": container with ID starting with ec90c05704989f88f4426dd5f25430ddaff75b8e4e1f05fe1652f1f3de28ead5 not found: ID does not exist" containerID="ec90c05704989f88f4426dd5f25430ddaff75b8e4e1f05fe1652f1f3de28ead5" Sep 05 08:23:14.354289 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:14.354251 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec90c05704989f88f4426dd5f25430ddaff75b8e4e1f05fe1652f1f3de28ead5"} err="failed to get container status \"ec90c05704989f88f4426dd5f25430ddaff75b8e4e1f05fe1652f1f3de28ead5\": rpc error: code = NotFound desc = could not find container \"ec90c05704989f88f4426dd5f25430ddaff75b8e4e1f05fe1652f1f3de28ead5\": container with ID starting with ec90c05704989f88f4426dd5f25430ddaff75b8e4e1f05fe1652f1f3de28ead5 not found: ID does not exist" Sep 05 08:23:14.359322 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:14.359300 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw"] Sep 05 08:23:14.363163 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:14.363139 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7548c976dd-f7ggw"] Sep 05 08:23:15.886200 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:15.886164 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd1dd31-3567-42ce-9f58-282db972f7b0" path="/var/lib/kubelet/pods/9cd1dd31-3567-42ce-9f58-282db972f7b0/volumes" Sep 05 08:23:18.052294 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:18.052270 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 08:23:18.055191 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:18.055168 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 08:23:18.055595 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:18.055575 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 08:23:18.058399 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:18.058380 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zbn86_b3dc39b8-d4d0-469d-8d91-f7d4ed7265cb/ovn-acl-logging/0.log" Sep 05 08:23:21.246885 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:21.246849 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:23:29.348151 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:29.348105 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:23:37.540462 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:37.540426 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-dx9qf"] Sep 05 08:23:45.109127 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:45.109095 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9fc4f56d7-lnd8l_a545c090-555e-45cd-ac7d-6e9e3625aaf6/copy-manifests/0.log" Sep 05 08:23:45.115906 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:45.115872 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9fc4f56d7-lnd8l_a545c090-555e-45cd-ac7d-6e9e3625aaf6/manager/0.log" Sep 05 08:23:45.692299 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:45.692264 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7895c5cf96-9zr5g_bc73b286-ca2a-42b3-bc32-21ba59b095a6/maas-api/0.log" Sep 05 08:23:45.919395 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:45.919368 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29809920-zn8n8_74c20fd9-95e6-46ad-93a4-5021d1d39caa/cleanup/0.log" Sep 05 08:23:47.073342 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:47.073309 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-czbkm_cdb06401-71c2-4153-b4ac-18cd7bda3f4a/manager/0.log" Sep 05 08:23:47.179281 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:47.179254 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-x262b_5b7a0023-07e1-4c29-9919-9aed2462f386/manager/0.log" Sep 05 08:23:47.291548 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:47.291511 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vxbgr_b2098fd7-17ed-4507-81c4-7c889eba0659/kuadrant-console-plugin/0.log" Sep 05 08:23:47.420820 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:47.420722 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-ps5q6_b73aad89-bbde-488e-8d5d-5866a5d6f4c8/manager/0.log" Sep 05 08:23:47.552996 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:47.552969 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-dx9qf_91e7d7f1-b79c-4df5-b679-0a2310994202/limitador/0.log" Sep 05 08:23:48.146644 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:48.146616 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-v8mm4_607f4444-997d-48a8-81d7-015547d579f6/discovery/0.log" Sep 05 08:23:48.257304 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:48.257280 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74c76cbcf8-6mjdg_66cc75f7-944e-4f09-83ba-0e2c5f83221b/kube-auth-proxy/0.log" Sep 05 08:23:49.492045 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:49.492013 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-b659b_87f228a1-6d98-4c4e-b59d-f83d249b50e8/main/0.log" Sep 05 08:23:56.103029 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.102991 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-24lm5/must-gather-ms79p"] Sep 05 08:23:56.103442 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.103329 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd1dd31-3567-42ce-9f58-282db972f7b0" containerName="maas-api" Sep 05 08:23:56.103442 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.103339 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd1dd31-3567-42ce-9f58-282db972f7b0" containerName="maas-api" Sep 05 08:23:56.103442 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.103393 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd1dd31-3567-42ce-9f58-282db972f7b0" containerName="maas-api" Sep 05 08:23:56.105896 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.105881 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-24lm5/must-gather-ms79p" Sep 05 08:23:56.107972 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.107945 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-24lm5\"/\"openshift-service-ca.crt\"" Sep 05 08:23:56.108111 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.107978 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-24lm5\"/\"default-dockercfg-z7vr5\"" Sep 05 08:23:56.108111 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.107978 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-24lm5\"/\"kube-root-ca.crt\"" Sep 05 08:23:56.116362 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.116338 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-24lm5/must-gather-ms79p"] Sep 05 08:23:56.277538 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.277497 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4cdd1b3-a793-4bd8-8905-c73c59c4b413-must-gather-output\") pod \"must-gather-ms79p\" (UID: \"a4cdd1b3-a793-4bd8-8905-c73c59c4b413\") " pod="openshift-must-gather-24lm5/must-gather-ms79p" Sep 05 08:23:56.277538 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.277548 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjntb\" (UniqueName: \"kubernetes.io/projected/a4cdd1b3-a793-4bd8-8905-c73c59c4b413-kube-api-access-hjntb\") pod \"must-gather-ms79p\" (UID: \"a4cdd1b3-a793-4bd8-8905-c73c59c4b413\") " pod="openshift-must-gather-24lm5/must-gather-ms79p" Sep 05 08:23:56.378239 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.378134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4cdd1b3-a793-4bd8-8905-c73c59c4b413-must-gather-output\") pod \"must-gather-ms79p\" (UID: \"a4cdd1b3-a793-4bd8-8905-c73c59c4b413\") " pod="openshift-must-gather-24lm5/must-gather-ms79p" Sep 05 08:23:56.378239 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.378176 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjntb\" (UniqueName: \"kubernetes.io/projected/a4cdd1b3-a793-4bd8-8905-c73c59c4b413-kube-api-access-hjntb\") pod \"must-gather-ms79p\" (UID: \"a4cdd1b3-a793-4bd8-8905-c73c59c4b413\") " pod="openshift-must-gather-24lm5/must-gather-ms79p" Sep 05 08:23:56.378573 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.378550 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4cdd1b3-a793-4bd8-8905-c73c59c4b413-must-gather-output\") pod \"must-gather-ms79p\" (UID: \"a4cdd1b3-a793-4bd8-8905-c73c59c4b413\") " pod="openshift-must-gather-24lm5/must-gather-ms79p" Sep 05 08:23:56.385394 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.385371 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjntb\" (UniqueName: \"kubernetes.io/projected/a4cdd1b3-a793-4bd8-8905-c73c59c4b413-kube-api-access-hjntb\") pod \"must-gather-ms79p\" (UID: \"a4cdd1b3-a793-4bd8-8905-c73c59c4b413\") " pod="openshift-must-gather-24lm5/must-gather-ms79p" Sep 05 08:23:56.415098 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.415071 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-24lm5/must-gather-ms79p" Sep 05 08:23:56.685668 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:56.685644 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-24lm5/must-gather-ms79p"] Sep 05 08:23:56.688335 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:23:56.688308 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4cdd1b3_a793_4bd8_8905_c73c59c4b413.slice/crio-e38f596e88ddad8dc7ce6bb14438080d70ded61064c6f86c11f27fb617182572 WatchSource:0}: Error finding container e38f596e88ddad8dc7ce6bb14438080d70ded61064c6f86c11f27fb617182572: Status 404 returned error can't find the container with id e38f596e88ddad8dc7ce6bb14438080d70ded61064c6f86c11f27fb617182572 Sep 05 08:23:57.574593 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:57.574563 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-24lm5/must-gather-ms79p" event={"ID":"a4cdd1b3-a793-4bd8-8905-c73c59c4b413","Type":"ContainerStarted","Data":"e38f596e88ddad8dc7ce6bb14438080d70ded61064c6f86c11f27fb617182572"} Sep 05 08:23:58.585935 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:58.585894 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-24lm5/must-gather-ms79p" event={"ID":"a4cdd1b3-a793-4bd8-8905-c73c59c4b413","Type":"ContainerStarted","Data":"aaedc67e1ff0f22a1384f8b154eb2090e41ef6ad8dad2c35336c9d77c6b30634"} Sep 05 08:23:58.586444 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:58.586277 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-24lm5/must-gather-ms79p" event={"ID":"a4cdd1b3-a793-4bd8-8905-c73c59c4b413","Type":"ContainerStarted","Data":"ce568511be188291d28845ac7a57da3186ed63ab2c6aa645336840c955429fb5"} Sep 05 08:23:58.601832 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:58.601775 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-24lm5/must-gather-ms79p" podStartSLOduration=1.789520247 podStartE2EDuration="2.601753857s" totalImagesPullingTime="812.23361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-05 08:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:23:58.599688873 +0000 UTC m=+2141.208663998" watchObservedRunningTime="2026-09-05 08:23:58.601753857 +0000 UTC m=+2141.210728981" Sep 05 08:23:59.131519 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:59.131483 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xtq7n_b43f76a2-cd92-49a6-b567-71f3ad019d30/global-pull-secret-syncer/0.log" Sep 05 08:23:59.238333 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:59.238305 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xpmql_dea8be0d-b3bc-4bb9-80d5-00bf1d08a034/konnectivity-agent/0.log" Sep 05 08:23:59.306623 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:23:59.306592 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-184.ec2.internal_76bb8a07519e0ff54e97f8d73d9e9f48/haproxy/0.log" Sep 05 08:24:04.165287 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:04.165256 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-czbkm_cdb06401-71c2-4153-b4ac-18cd7bda3f4a/manager/0.log" Sep 05 08:24:04.194243 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:04.194186 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-x262b_5b7a0023-07e1-4c29-9919-9aed2462f386/manager/0.log" Sep 05 08:24:04.224933 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:04.224840 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-vxbgr_b2098fd7-17ed-4507-81c4-7c889eba0659/kuadrant-console-plugin/0.log" Sep 05 08:24:04.271091 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:04.271053 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-ps5q6_b73aad89-bbde-488e-8d5d-5866a5d6f4c8/manager/0.log" Sep 05 08:24:04.303725 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:04.303699 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-dx9qf_91e7d7f1-b79c-4df5-b679-0a2310994202/limitador/0.log" Sep 05 08:24:05.999469 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:05.999432 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-npnpg_c0bd8583-070e-4b16-96bf-3780bbb5497f/cluster-monitoring-operator/0.log" Sep 05 08:24:06.328273 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:06.328235 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zx7wd_58a109f7-4b95-4432-9289-ce6e72e852f6/node-exporter/0.log" Sep 05 08:24:06.351561 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:06.351528 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zx7wd_58a109f7-4b95-4432-9289-ce6e72e852f6/kube-rbac-proxy/0.log" Sep 05 08:24:06.375424 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:06.375392 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zx7wd_58a109f7-4b95-4432-9289-ce6e72e852f6/init-textfile/0.log" Sep 05 08:24:09.131647 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.131544 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/1.log" Sep 05 08:24:09.138193 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.138165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-hhvm4_3da53738-9ae4-473e-b55a-d7379875a4a6/console-operator/2.log" Sep 05 08:24:09.347080 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.347045 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58"] Sep 05 08:24:09.355043 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.355004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.360125 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.360092 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58"] Sep 05 08:24:09.410007 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.409918 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27284dc0-0316-49ac-9c07-92c1c0a75369-podres\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.410349 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.410322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27284dc0-0316-49ac-9c07-92c1c0a75369-sys\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.410582 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.410563 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jvv\" (UniqueName: \"kubernetes.io/projected/27284dc0-0316-49ac-9c07-92c1c0a75369-kube-api-access-s5jvv\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.410748 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.410731 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27284dc0-0316-49ac-9c07-92c1c0a75369-proc\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.410897 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.410882 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27284dc0-0316-49ac-9c07-92c1c0a75369-lib-modules\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.512423 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.512372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27284dc0-0316-49ac-9c07-92c1c0a75369-podres\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.512616 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.512436 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27284dc0-0316-49ac-9c07-92c1c0a75369-sys\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.512616 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.512508 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jvv\" (UniqueName: \"kubernetes.io/projected/27284dc0-0316-49ac-9c07-92c1c0a75369-kube-api-access-s5jvv\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.512616 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.512544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27284dc0-0316-49ac-9c07-92c1c0a75369-proc\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.512616 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.512575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27284dc0-0316-49ac-9c07-92c1c0a75369-lib-modules\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.512872 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.512845 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27284dc0-0316-49ac-9c07-92c1c0a75369-lib-modules\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.512976 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.512953 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/27284dc0-0316-49ac-9c07-92c1c0a75369-podres\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.513106 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.513006 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27284dc0-0316-49ac-9c07-92c1c0a75369-sys\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.513390 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.513368 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/27284dc0-0316-49ac-9c07-92c1c0a75369-proc\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.520465 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.520434 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jvv\" (UniqueName: \"kubernetes.io/projected/27284dc0-0316-49ac-9c07-92c1c0a75369-kube-api-access-s5jvv\") pod \"perf-node-gather-daemonset-5tx58\" (UID: \"27284dc0-0316-49ac-9c07-92c1c0a75369\") " pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.667914 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.667813 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:09.966103 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:09.965926 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58"] Sep 05 08:24:09.968723 ip-10-0-143-184 kubenswrapper[2576]: W0905 08:24:09.968687 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod27284dc0_0316_49ac_9c07_92c1c0a75369.slice/crio-578aaed44ba1476c9cb8ff9858d1c5a17b123ba6f73b6d2f3bbed857b83d364f WatchSource:0}: Error finding container 578aaed44ba1476c9cb8ff9858d1c5a17b123ba6f73b6d2f3bbed857b83d364f: Status 404 returned error can't find the container with id 578aaed44ba1476c9cb8ff9858d1c5a17b123ba6f73b6d2f3bbed857b83d364f Sep 05 08:24:10.683430 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:10.683383 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" event={"ID":"27284dc0-0316-49ac-9c07-92c1c0a75369","Type":"ContainerStarted","Data":"2e533821769608828844532ead2ddefa120246c96039e0075029aefa349acd79"} Sep 05 08:24:10.683961 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:10.683462 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" event={"ID":"27284dc0-0316-49ac-9c07-92c1c0a75369","Type":"ContainerStarted","Data":"578aaed44ba1476c9cb8ff9858d1c5a17b123ba6f73b6d2f3bbed857b83d364f"} Sep 05 08:24:10.683961 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:10.683643 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:10.700168 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:10.700107 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" podStartSLOduration=1.700085177 podStartE2EDuration="1.700085177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-05 08:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-05 08:24:10.696753281 +0000 UTC m=+2153.305728406" watchObservedRunningTime="2026-09-05 08:24:10.700085177 +0000 UTC m=+2153.309060301" Sep 05 08:24:11.526888 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:11.526856 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7cs9n_332ec72e-6421-48a2-b0c0-def5d946f80c/dns/0.log" Sep 05 08:24:11.551155 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:11.551122 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7cs9n_332ec72e-6421-48a2-b0c0-def5d946f80c/kube-rbac-proxy/0.log" Sep 05 08:24:11.669079 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:11.669051 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4bmbd_a6bf702d-539a-4164-9e92-18a93205e30f/dns-node-resolver/0.log" Sep 05 08:24:12.264390 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:12.264355 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-brf5d_13432308-4a00-44e2-babf-6daaff232a0a/node-ca/0.log" Sep 05 08:24:13.607828 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:13.607795 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-v8mm4_607f4444-997d-48a8-81d7-015547d579f6/discovery/0.log" Sep 05 08:24:13.629632 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:13.629603 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74c76cbcf8-6mjdg_66cc75f7-944e-4f09-83ba-0e2c5f83221b/kube-auth-proxy/0.log" Sep 05 08:24:14.680426 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:14.680397 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rbg4k_efa1fb29-89ac-4b0d-aa70-ab6e4a2ef359/serve-healthcheck-canary/0.log" Sep 05 08:24:15.162365 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:15.162336 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cf2bt_a2bc7982-908a-4621-b8d8-962a902ff34d/kube-rbac-proxy/0.log" Sep 05 08:24:15.185141 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:15.185114 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cf2bt_a2bc7982-908a-4621-b8d8-962a902ff34d/exporter/0.log" Sep 05 08:24:15.207074 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:15.207040 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-cf2bt_a2bc7982-908a-4621-b8d8-962a902ff34d/extractor/0.log" Sep 05 08:24:16.701984 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:16.701951 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-24lm5/perf-node-gather-daemonset-5tx58" Sep 05 08:24:17.757039 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:17.757012 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9fc4f56d7-lnd8l_a545c090-555e-45cd-ac7d-6e9e3625aaf6/manager/0.log" Sep 05 08:24:17.778186 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:17.778155 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9fc4f56d7-lnd8l_a545c090-555e-45cd-ac7d-6e9e3625aaf6/copy-manifests/0.log" Sep 05 08:24:18.300938 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:18.300903 2576 scope.go:117] "RemoveContainer" containerID="d8a00c2ed4ff3f4c4f769d5863a1070ac64cfd0511c8ac37b089765ced09faac" Sep 05 08:24:20.654837 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:20.654793 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-v8mm4_607f4444-997d-48a8-81d7-015547d579f6/discovery/0.log" Sep 05 08:24:20.665490 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:20.665465 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74c76cbcf8-6mjdg_66cc75f7-944e-4f09-83ba-0e2c5f83221b/kube-auth-proxy/0.log" Sep 05 08:24:22.649023 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:22.648991 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-v8mm4_607f4444-997d-48a8-81d7-015547d579f6/discovery/0.log" Sep 05 08:24:22.666284 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:22.666255 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-74c76cbcf8-6mjdg_66cc75f7-944e-4f09-83ba-0e2c5f83221b/kube-auth-proxy/0.log" Sep 05 08:24:24.656925 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:24.656892 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-m9n7b_6655c29a-17e5-4402-b6b1-4550f3ac99dc/migrator/0.log" Sep 05 08:24:24.678739 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:24.678710 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-m9n7b_6655c29a-17e5-4402-b6b1-4550f3ac99dc/graceful-termination/0.log" Sep 05 08:24:26.168935 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:26.168906 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dsknf_e1d05ed4-423c-4619-89c1-3d4a00b8ea86/kube-multus-additional-cni-plugins/0.log" Sep 05 08:24:26.191264 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:26.191237 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dsknf_e1d05ed4-423c-4619-89c1-3d4a00b8ea86/egress-router-binary-copy/0.log" Sep 05 08:24:26.213894 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:26.213870 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dsknf_e1d05ed4-423c-4619-89c1-3d4a00b8ea86/cni-plugins/0.log" Sep 05 08:24:26.240302 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:26.240280 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dsknf_e1d05ed4-423c-4619-89c1-3d4a00b8ea86/bond-cni-plugin/0.log" Sep 05 08:24:26.262006 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:26.261976 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dsknf_e1d05ed4-423c-4619-89c1-3d4a00b8ea86/routeoverride-cni/0.log" Sep 05 08:24:26.284328 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:26.284298 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dsknf_e1d05ed4-423c-4619-89c1-3d4a00b8ea86/whereabouts-cni-bincopy/0.log" Sep 05 08:24:26.306011 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:26.305972 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dsknf_e1d05ed4-423c-4619-89c1-3d4a00b8ea86/whereabouts-cni/0.log" Sep 05 08:24:26.556631 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:26.556596 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l224c_861a7bc4-b5ce-4432-9ba4-c2f938aab3f3/kube-multus/0.log" Sep 05 08:24:26.673660 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:26.673575 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zr6kd_73836491-7624-4e24-a8cd-2ad60bc09155/network-metrics-daemon/0.log" Sep 05 08:24:26.692427 ip-10-0-143-184 kubenswrapper[2576]: I0905 08:24:26.692398 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zr6kd_73836491-7624-4e24-a8cd-2ad60bc09155/kube-rbac-proxy/0.log"