Sep 10 19:13:39.979745 ip-10-0-138-71 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 10 19:13:39.979756 ip-10-0-138-71 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 10 19:13:39.979765 ip-10-0-138-71 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 10 19:13:39.979977 ip-10-0-138-71 systemd[1]: Failed to start Kubernetes Kubelet. Sep 10 19:13:50.114478 ip-10-0-138-71 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 10 19:13:50.114501 ip-10-0-138-71 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 69314fe8f11b472a9840595e36bd450f -- Sep 10 19:15:49.542753 ip-10-0-138-71 systemd[1]: Starting Kubernetes Kubelet... Sep 10 19:15:49.883528 ip-10-0-138-71 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 19:15:49.883528 ip-10-0-138-71 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 19:15:49.883528 ip-10-0-138-71 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 19:15:49.883528 ip-10-0-138-71 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 19:15:49.883528 ip-10-0-138-71 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 19:15:49.884533 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.884434 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 19:15:49.887792 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887772 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:15:49.887792 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887790 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:15:49.887792 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887796 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887802 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887807 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887810 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887814 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887818 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887821 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887824 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887828 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887832 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887836 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887839 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887845 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887850 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887855 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887860 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887865 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887869 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887873 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887877 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:15:49.887981 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887881 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887885 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887889 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887893 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887897 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887901 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887905 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887909 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887913 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887917 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887921 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887926 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887932 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887937 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887941 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887945 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887950 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887954 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887959 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887963 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:15:49.888807 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887969 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887973 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887977 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887982 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887985 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887989 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887994 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.887998 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888002 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888006 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888010 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888016 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888022 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888027 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888032 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888036 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888041 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888045 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888049 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888054 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:15:49.889607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888058 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888062 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888065 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888070 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888076 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888080 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888085 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888089 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888095 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888099 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888104 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888108 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888112 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888116 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888120 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888124 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888128 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888132 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888136 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:15:49.890090 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888140 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888145 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888149 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888154 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888158 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888162 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888166 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888170 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888174 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888178 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888182 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888186 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888190 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888194 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888198 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888202 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888207 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:15:49.890584 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.888211 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:15:49.892411 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892394 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:15:49.892411 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892410 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:15:49.892411 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892413 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892417 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892420 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892424 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892427 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892429 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892432 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892434 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892438 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892443 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892448 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892451 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892454 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892457 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892459 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892462 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892464 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892467 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892469 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:15:49.892574 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892491 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892494 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892497 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892499 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892502 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892505 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892508 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892511 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892515 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892517 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892520 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892523 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892526 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892529 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892532 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892534 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892537 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892540 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892542 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892545 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:15:49.893089 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892547 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892550 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892552 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892554 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892557 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892560 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892562 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892564 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892567 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892571 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892573 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892577 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892580 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892582 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892585 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892587 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892590 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892593 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892595 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892598 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:15:49.893607 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892600 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892603 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892605 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892608 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892610 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892613 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892616 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892618 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892621 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892623 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892626 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892629 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892631 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892634 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892636 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892639 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892642 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892644 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892647 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:15:49.894106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892650 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892654 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892657 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892660 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892662 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892665 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892667 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892669 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892672 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892674 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892677 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892679 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892681 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892684 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892687 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892689 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892691 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892694 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.892697 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893073 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:15:49.894591 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893078 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893080 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893083 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893085 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893088 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893091 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893094 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893097 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893099 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893102 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893105 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893107 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893110 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893112 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893116 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893120 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893123 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893126 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893129 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893132 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:15:49.895082 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893134 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893137 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893139 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893142 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893144 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893147 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893149 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893152 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893154 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893156 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893159 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893162 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893165 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893167 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893169 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893172 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893174 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893177 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893180 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893182 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:15:49.895593 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893185 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893188 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893190 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893193 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893195 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893197 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893200 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893202 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893205 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893208 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893212 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893215 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893217 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893219 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893222 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893224 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893227 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893229 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893232 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:15:49.896095 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893234 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893236 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893239 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893241 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893244 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893247 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893249 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893252 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893254 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893256 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893260 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893262 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893265 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893267 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893270 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893272 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893275 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893277 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893279 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893282 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:15:49.896564 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893284 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893287 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893292 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893295 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893297 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893299 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893302 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893304 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893307 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893309 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893312 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893314 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893316 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893319 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893321 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893324 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893327 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893330 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.893334 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893414 2578 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893453 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 19:15:49.897052 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893460 2578 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893464 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893468 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893489 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893494 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893497 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893500 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893503 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893506 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893509 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893512 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893515 2578 flags.go:64] FLAG: --cgroup-root="" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893518 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893521 2578 flags.go:64] FLAG: --client-ca-file="" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893525 2578 flags.go:64] FLAG: --cloud-provider="external" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893528 2578 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893533 2578 flags.go:64] FLAG: --cluster-domain="" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893536 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893539 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893542 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893545 2578 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893548 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893551 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 19:15:49.897612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893554 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893557 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893560 2578 flags.go:64] FLAG: --contention-profiling="false" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893563 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893566 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893569 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893572 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893575 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893579 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893581 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893584 2578 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893587 2578 flags.go:64] FLAG: --enable-server="true" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893590 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893593 2578 flags.go:64] FLAG: --event-burst="100" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893597 2578 flags.go:64] FLAG: --event-qps="50" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893600 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893602 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893605 2578 flags.go:64] FLAG: --eviction-hard="" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893608 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893611 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893614 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893617 2578 flags.go:64] FLAG: --eviction-soft="" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893620 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893623 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893626 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 19:15:49.898184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893629 2578 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893632 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893635 2578 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893637 2578 flags.go:64] FLAG: --feature-gates="" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893641 2578 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893644 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893647 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893650 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893653 2578 flags.go:64] FLAG: --healthz-port="10248" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893655 2578 flags.go:64] FLAG: --help="false" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893658 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-138-71.ec2.internal" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893661 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893663 2578 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893666 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893669 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893672 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893676 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893678 2578 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893681 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893684 2578 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893686 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893689 2578 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893692 2578 flags.go:64] FLAG: --kube-reserved="" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893695 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 19:15:49.898824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893697 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893700 2578 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893702 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893705 2578 flags.go:64] FLAG: --lock-file="" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893708 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893710 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893713 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893718 2578 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893721 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893724 2578 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893726 2578 flags.go:64] FLAG: --logging-format="text" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893729 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893732 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893734 2578 flags.go:64] FLAG: --manifest-url="" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893737 2578 flags.go:64] FLAG: --manifest-url-header="" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893741 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893744 2578 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893748 2578 flags.go:64] FLAG: --max-pods="110" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893751 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893754 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893756 2578 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893759 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893762 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893764 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893767 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 19:15:49.899406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893773 2578 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893776 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893779 2578 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893782 2578 flags.go:64] FLAG: --pod-cidr="" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893784 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893790 2578 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893792 2578 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893796 2578 flags.go:64] FLAG: --pods-per-core="0" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893798 2578 flags.go:64] FLAG: --port="10250" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893801 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893804 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01e34b94b925a1656" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893807 2578 flags.go:64] FLAG: --qos-reserved="" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893810 2578 flags.go:64] FLAG: --read-only-port="10255" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893812 2578 flags.go:64] FLAG: --register-node="true" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893815 2578 flags.go:64] FLAG: --register-with-taints="" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893818 2578 flags.go:64] FLAG: --registry-burst="10" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893822 2578 flags.go:64] FLAG: --registry-qps="5" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893825 2578 flags.go:64] FLAG: --reserved-cpus="" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893827 2578 flags.go:64] FLAG: --reserved-memory="" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893833 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893839 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893842 2578 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893845 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893847 2578 flags.go:64] FLAG: --runonce="false" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893850 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 19:15:49.900027 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893853 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893856 2578 flags.go:64] FLAG: --seccomp-default="false" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893859 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893861 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893864 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893867 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893870 2578 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893873 2578 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893876 2578 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893879 2578 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893881 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893884 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893887 2578 flags.go:64] FLAG: --system-cgroups="" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893890 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893895 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893898 2578 flags.go:64] FLAG: --tls-cert-file="" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893900 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893905 2578 flags.go:64] FLAG: --tls-min-version="" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893907 2578 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893910 2578 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893912 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893915 2578 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893918 2578 flags.go:64] FLAG: --v="2" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893921 2578 flags.go:64] FLAG: --version="false" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893926 2578 flags.go:64] FLAG: --vmodule="" Sep 10 19:15:49.900672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893929 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.893934 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894045 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894049 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894052 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894055 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894058 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894061 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894064 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894067 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894069 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894072 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894075 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894077 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894079 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894082 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894084 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894087 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894089 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894091 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:15:49.901325 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894094 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894096 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894098 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894101 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894103 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894106 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894108 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894111 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894113 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894116 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894118 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894121 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894124 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894128 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894132 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894134 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894137 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894139 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894142 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894144 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:15:49.901829 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894147 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894149 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894152 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894154 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894157 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894159 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894161 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894164 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894166 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894169 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894171 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894174 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894176 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894178 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894181 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894183 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894186 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894189 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894191 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:15:49.902326 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894195 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894199 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894202 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894206 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894209 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894212 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894218 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894231 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894235 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894238 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894241 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894243 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894246 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894248 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894251 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894253 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894256 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894258 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894261 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:15:49.902878 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894263 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894266 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894269 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894271 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894274 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894276 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894279 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894281 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894284 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894286 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894288 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894291 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894293 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894296 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894298 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894301 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894303 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894305 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894308 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894313 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:15:49.903348 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894317 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894319 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.894322 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.894831 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.901896 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.902020 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.903339 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.903350 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903414 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903420 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903424 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903427 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903430 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903433 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903435 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:15:49.903856 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903438 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903440 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903443 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903445 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903448 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903451 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903454 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903456 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903459 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903461 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903464 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903466 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903469 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903487 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903490 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903492 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903495 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903497 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903500 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:15:49.904237 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903502 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903505 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903508 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903512 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903514 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903517 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903520 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903523 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903526 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903528 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903530 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903533 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903535 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903538 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903542 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903544 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903546 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903549 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903551 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903554 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:15:49.904742 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903556 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903559 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903561 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903564 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903566 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903569 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903571 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903573 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903576 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903578 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903581 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903583 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903585 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903588 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903591 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903594 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903597 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903600 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903602 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903605 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:15:49.905230 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903608 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903612 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903616 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903619 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903622 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903624 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903627 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903630 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903632 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903635 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903637 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903640 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903642 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903646 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903650 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903653 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903655 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903657 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903660 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:15:49.905733 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903663 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903666 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903668 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903670 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903673 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903676 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903678 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903681 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903683 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903686 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903689 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903691 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903694 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903698 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.903702 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 19:15:49.906189 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903825 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903830 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903833 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903835 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903838 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903840 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903843 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903845 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903848 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903850 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903852 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903855 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903857 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903860 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903862 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903865 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903868 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903872 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903876 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903879 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 19:15:49.906662 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903881 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903884 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903887 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903890 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903892 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903895 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903897 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903900 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903903 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903905 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903908 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903910 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903912 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903915 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903917 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903919 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903922 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903924 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903926 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903929 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 19:15:49.907136 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903931 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903933 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903936 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903938 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903941 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903953 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903956 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903959 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903961 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903964 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903967 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903969 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903972 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903975 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903977 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903980 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903982 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903985 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903988 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 19:15:49.907687 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903991 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903993 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903996 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.903999 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904001 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904004 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904006 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904009 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904011 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904014 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904017 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904020 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904024 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904026 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904029 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904032 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904034 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904037 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904039 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 19:15:49.908146 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904042 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904045 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904049 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904051 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904053 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904056 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904058 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904061 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904063 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904065 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904068 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904070 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904073 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904075 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904077 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904080 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904089 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904091 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904094 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 19:15:49.908731 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904097 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 19:15:49.909275 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:49.904099 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 19:15:49.909275 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.904104 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 19:15:49.909275 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.904208 2578 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 19:15:49.909275 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.907403 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 19:15:49.909275 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.908186 2578 server.go:1019] "Starting client certificate rotation" Sep 10 19:15:49.909275 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.908300 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 19:15:49.909275 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.908344 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 19:15:49.910183 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.910156 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 19:15:49.911437 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.911421 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 19:15:49.922522 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.922499 2578 log.go:25] "Validated CRI v1 runtime API" Sep 10 19:15:49.925230 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.925215 2578 log.go:25] "Validated CRI v1 image API" Sep 10 19:15:49.926013 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.925996 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 19:15:49.928507 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.928491 2578 fs.go:135] Filesystem UUIDs: map[4e769f26-916b-4b59-aad5-6f41b86dcc3f:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 eaae758e-da46-4ce7-9064-7abde20d386d:/dev/nvme0n1p3] Sep 10 19:15:49.928548 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.928508 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 19:15:49.933280 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.933175 2578 manager.go:216] Machine: {Timestamp:2026-09-10 19:15:49.932097476 +0000 UTC m=+0.288849792 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109058 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec210666545fcdd308794b83cdef7ef6 SystemUUID:ec210666-545f-cdd3-0879-4b83cdef7ef6 BootID:69314fe8-f11b-472a-9840-595e36bd450f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:06:c5:48:88:e9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:06:c5:48:88:e9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:e3:24:47:39:56 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 19:15:49.933280 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.933275 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 19:15:49.933401 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.933374 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 19:15:49.934261 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.934241 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 19:15:49.934392 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.934263 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-71.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 19:15:49.935016 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.935006 2578 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 19:15:49.935052 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.935019 2578 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 19:15:49.935052 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.935026 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 19:15:49.935510 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.935500 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 19:15:49.936754 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.936745 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 19:15:49.939804 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.939792 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 10 19:15:49.940050 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.940040 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 19:15:49.940087 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.940079 2578 kubelet.go:486] "Attempting to sync node with API server" Sep 10 19:15:49.940118 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.940093 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 19:15:49.940145 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.940130 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 19:15:49.940172 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.940148 2578 kubelet.go:398] "Adding apiserver pod source" Sep 10 19:15:49.940172 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.940160 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 19:15:49.940545 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.940534 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 19:15:49.940597 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.940553 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 19:15:49.942290 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.942273 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 19:15:49.944563 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.944549 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 19:15:49.944611 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.944579 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 19:15:49.944727 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.944717 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 19:15:49.944764 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.944731 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 19:15:49.944764 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.944737 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 19:15:49.944764 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.944742 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 19:15:49.944764 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.944749 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 19:15:49.944764 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.944754 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 19:15:49.945187 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.945178 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 19:15:49.945229 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.945189 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 19:15:49.945229 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.945196 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 19:15:49.945229 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.945201 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 19:15:49.945229 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.945207 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 19:15:49.945229 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.945213 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 19:15:49.945378 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.945240 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 19:15:49.945817 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.945809 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 19:15:49.945817 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.945817 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 19:15:49.950213 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.950195 2578 server.go:1268] "Started kubelet" Sep 10 19:15:49.950499 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.950348 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 19:15:49.950499 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.950388 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 19:15:49.950499 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.950453 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 19:15:49.951047 ip-10-0-138-71 systemd[1]: Started Kubernetes Kubelet. Sep 10 19:15:49.951684 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.951665 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 19:15:49.952545 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.952532 2578 server.go:310] "Adding debug handlers to kubelet server" Sep 10 19:15:49.953256 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.953238 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-71.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 19:15:49.953341 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:49.953310 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 19:15:49.953341 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:49.953328 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 19:15:49.954669 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:49.954647 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 19:15:49.955638 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.955626 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 19:15:49.955697 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.955680 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 19:15:49.955758 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:49.955740 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-71.ec2.internal\" not found" Sep 10 19:15:49.956351 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.956332 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 19:15:49.956450 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.956354 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 19:15:49.956450 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.956340 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 19:15:49.956582 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.956467 2578 reconstruct.go:97] "Volume reconstruction finished" Sep 10 19:15:49.956582 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.956493 2578 reconciler.go:29] "Reconciler: start to sync state" Sep 10 19:15:49.956582 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.956496 2578 factory.go:55] Registering systemd factory Sep 10 19:15:49.956582 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.956532 2578 factory.go:223] Registration of the systemd container factory successfully Sep 10 19:15:49.956847 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.956829 2578 factory.go:153] Registering CRI-O factory Sep 10 19:15:49.956937 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.956847 2578 factory.go:223] Registration of the crio container factory successfully Sep 10 19:15:49.956937 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.956903 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 19:15:49.956937 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.956916 2578 factory.go:103] Registering Raw factory Sep 10 19:15:49.956937 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.956927 2578 manager.go:1196] Started watching for new ooms in manager Sep 10 19:15:49.957273 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.957259 2578 manager.go:349] Starting recovery of all containers Sep 10 19:15:49.963794 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:49.963762 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 10 19:15:49.963897 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:49.963779 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-71.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 10 19:15:49.967674 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:49.963884 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-71.ec2.internal.18d40ba3c2e3088e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-71.ec2.internal,UID:ip-10-0-138-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-71.ec2.internal,},FirstTimestamp:2026-09-10 19:15:49.950163086 +0000 UTC m=+0.306915402,LastTimestamp:2026-09-10 19:15:49.950163086 +0000 UTC m=+0.306915402,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-71.ec2.internal,}" Sep 10 19:15:49.971329 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.971302 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8vj5v" Sep 10 19:15:49.974183 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.974162 2578 manager.go:354] Recovery completed Sep 10 19:15:49.978162 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.978150 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:15:49.979611 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.979594 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8vj5v" Sep 10 19:15:49.979893 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.979874 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:15:49.979974 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.979908 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:15:49.979974 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.979921 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:15:49.980467 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.980450 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 19:15:49.980467 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.980468 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 19:15:49.980589 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.980499 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:15:49.980589 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.980502 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 10 19:15:49.981186 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.981171 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:15:49.981262 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.981200 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:15:49.981262 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.981210 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:15:49.981721 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.981710 2578 policy_none.go:49] "None policy: Start" Sep 10 19:15:49.981758 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.981726 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 19:15:49.981758 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.981735 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 19:15:49.982122 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:49.982064 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-71.ec2.internal.18d40ba3c4a8a7cb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-71.ec2.internal,UID:ip-10-0-138-71.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-71.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-71.ec2.internal,},FirstTimestamp:2026-09-10 19:15:49.979891659 +0000 UTC m=+0.336643975,LastTimestamp:2026-09-10 19:15:49.979891659 +0000 UTC m=+0.336643975,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-71.ec2.internal,}" Sep 10 19:15:49.982797 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:49.982782 2578 policy_none.go:47] "Start" Sep 10 19:15:50.033033 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.033011 2578 manager.go:337] "Starting Device Plugin manager" Sep 10 19:15:50.035819 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.033055 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 19:15:50.035819 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.033068 2578 server.go:87] "Starting device plugin registration server" Sep 10 19:15:50.035819 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.033308 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 19:15:50.035819 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.033318 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 19:15:50.035819 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.033401 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 19:15:50.035819 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.033488 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 19:15:50.035819 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.033497 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 19:15:50.035819 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.034034 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 19:15:50.035819 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.034076 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-71.ec2.internal\" not found" Sep 10 19:15:50.133782 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.133708 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:15:50.134588 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.134563 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:15:50.134690 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.134596 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:15:50.134690 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.134608 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:15:50.134690 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.134631 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.144638 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.144621 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.144694 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.144639 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-71.ec2.internal\": node \"ip-10-0-138-71.ec2.internal\" not found" Sep 10 19:15:50.156437 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.156404 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 19:15:50.157737 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.157719 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 19:15:50.157795 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.157775 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 19:15:50.157839 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.157799 2578 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 19:15:50.157889 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.157842 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 19:15:50.160584 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.160567 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:15:50.162272 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.162258 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-71.ec2.internal\" not found" Sep 10 19:15:50.258296 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.258261 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-71.ec2.internal"] Sep 10 19:15:50.258396 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.258331 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:15:50.259304 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.259288 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:15:50.259382 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.259321 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:15:50.259382 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.259334 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:15:50.260883 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.260871 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:15:50.261008 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.260995 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.261046 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.261022 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:15:50.261619 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.261604 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:15:50.261696 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.261631 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:15:50.261696 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.261646 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:15:50.261696 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.261608 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:15:50.261794 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.261733 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:15:50.261794 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.261748 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:15:50.262367 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.262355 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-71.ec2.internal\" not found" Sep 10 19:15:50.263780 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.263766 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.263831 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.263794 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 19:15:50.264489 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.264453 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientMemory" Sep 10 19:15:50.264552 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.264499 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 19:15:50.264552 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.264513 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeHasSufficientPID" Sep 10 19:15:50.282074 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.282052 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-71.ec2.internal\" not found" node="ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.285348 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.285332 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-71.ec2.internal\" not found" node="ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.358837 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.358814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a79495c71582436fce13ab595ab2b7e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal\" (UID: \"a79495c71582436fce13ab595ab2b7e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.358936 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.358845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a79495c71582436fce13ab595ab2b7e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal\" (UID: \"a79495c71582436fce13ab595ab2b7e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.358936 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.358869 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d41321aee93f122eed6c1597c05c23f6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-71.ec2.internal\" (UID: \"d41321aee93f122eed6c1597c05c23f6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.363272 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.363256 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-71.ec2.internal\" not found" Sep 10 19:15:50.459982 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.459908 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a79495c71582436fce13ab595ab2b7e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal\" (UID: \"a79495c71582436fce13ab595ab2b7e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.459982 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.459955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a79495c71582436fce13ab595ab2b7e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal\" (UID: \"a79495c71582436fce13ab595ab2b7e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.459982 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.459977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d41321aee93f122eed6c1597c05c23f6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-71.ec2.internal\" (UID: \"d41321aee93f122eed6c1597c05c23f6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.460158 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.459999 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a79495c71582436fce13ab595ab2b7e7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal\" (UID: \"a79495c71582436fce13ab595ab2b7e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.460158 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.460021 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a79495c71582436fce13ab595ab2b7e7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal\" (UID: \"a79495c71582436fce13ab595ab2b7e7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.460158 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.460026 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d41321aee93f122eed6c1597c05c23f6-config\") pod \"kube-apiserver-proxy-ip-10-0-138-71.ec2.internal\" (UID: \"d41321aee93f122eed6c1597c05c23f6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.464018 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.464006 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-71.ec2.internal\" not found" Sep 10 19:15:50.564600 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.564563 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-71.ec2.internal\" not found" Sep 10 19:15:50.585732 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.585712 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.588661 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.588644 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-71.ec2.internal" Sep 10 19:15:50.664791 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.664763 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-71.ec2.internal\" not found" Sep 10 19:15:50.765564 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.765507 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-71.ec2.internal\" not found" Sep 10 19:15:50.768136 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.768115 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:15:50.866312 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.866282 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-71.ec2.internal\" not found" Sep 10 19:15:50.875215 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.875195 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:15:50.908540 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.908520 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 19:15:50.908931 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.908646 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 19:15:50.908931 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.908646 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 19:15:50.908931 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.908646 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 19:15:50.966400 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:50.966369 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-71.ec2.internal\" not found" Sep 10 19:15:50.981784 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.981751 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 19:10:51 +0000 UTC" deadline="2028-02-23 12:53:22.961628717 +0000 UTC" Sep 10 19:15:50.981784 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:50.981776 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12737h37m31.979855258s" Sep 10 19:15:51.067459 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:51.067435 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-71.ec2.internal\" not found" Sep 10 19:15:51.120061 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:51.119893 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:15:51.156795 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:51.156770 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal" Sep 10 19:15:51.167368 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:51.167348 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 19:15:51.168197 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:51.168184 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-71.ec2.internal" Sep 10 19:15:51.176565 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:51.176541 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 19:15:51.181916 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:51.181886 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd41321aee93f122eed6c1597c05c23f6.slice/crio-9af7986498b1334f3de2886774a340b8ed17af6956da01d36b31f376a7ffd273 WatchSource:0}: Error finding container 9af7986498b1334f3de2886774a340b8ed17af6956da01d36b31f376a7ffd273: Status 404 returned error can't find the container with id 9af7986498b1334f3de2886774a340b8ed17af6956da01d36b31f376a7ffd273 Sep 10 19:15:51.182435 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:51.182416 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79495c71582436fce13ab595ab2b7e7.slice/crio-148866d357883d9668244ed4b9a40f68bc10e0a65eb2e37c5ce328612cc3f0b2 WatchSource:0}: Error finding container 148866d357883d9668244ed4b9a40f68bc10e0a65eb2e37c5ce328612cc3f0b2: Status 404 returned error can't find the container with id 148866d357883d9668244ed4b9a40f68bc10e0a65eb2e37c5ce328612cc3f0b2 Sep 10 19:15:51.187212 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:51.187198 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 19:15:53.167661 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.167633 2578 apiserver.go:52] "Watching apiserver" Sep 10 19:15:53.174033 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.174008 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 19:15:53.175119 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.175092 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-bzjwd","kube-system/kube-apiserver-proxy-ip-10-0-138-71.ec2.internal","openshift-cluster-node-tuning-operator/tuned-q2zsb","openshift-dns/node-resolver-nqdhp","openshift-image-registry/node-ca-4b2dk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal","openshift-multus/multus-additional-cni-plugins-xcrxb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq","openshift-multus/multus-rt7fj","openshift-multus/network-metrics-daemon-lr779","openshift-network-diagnostics/network-check-target-sc84z","openshift-network-operator/iptables-alerter-nxp9t","openshift-ovn-kubernetes/ovnkube-node-75g24","kube-system/global-pull-secret-syncer-xdr42"] Sep 10 19:15:53.177613 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.176798 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.178998 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.178978 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.180527 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.180228 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-x9bc4\"" Sep 10 19:15:53.180527 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.180280 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 19:15:53.180527 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.180321 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 19:15:53.180527 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.180417 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 19:15:53.181005 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.180891 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 19:15:53.181715 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.181191 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 19:15:53.181715 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.181199 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 19:15:53.181715 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.181431 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.181715 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.181460 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vdpbj\"" Sep 10 19:15:53.182447 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.181948 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 19:15:53.182447 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.182007 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 19:15:53.182447 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.182197 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 19:15:53.182447 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.182220 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 19:15:53.182447 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.182242 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 19:15:53.186536 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.183594 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:15:53.186536 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.183678 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 19:15:53.186536 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.183776 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 19:15:53.186536 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.183856 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-5d8ss\"" Sep 10 19:15:53.186536 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.184268 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nqdhp" Sep 10 19:15:53.186536 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.185492 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4b2dk" Sep 10 19:15:53.186536 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.186471 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:53.187044 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.186549 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:15:53.188757 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.187498 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.188757 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.188703 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bzjwd" Sep 10 19:15:53.192023 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.191997 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.192222 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.192205 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 19:15:53.192562 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.192544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff5abec0-8905-4593-bebe-c10577813108-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.192684 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.192669 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-run-openvswitch\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.192802 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.192785 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-run\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.192867 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.192821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-sys\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.192867 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.192856 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff5abec0-8905-4593-bebe-c10577813108-cnibin\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.192965 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.192887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff5abec0-8905-4593-bebe-c10577813108-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.193024 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5510dd56-a971-468d-9909-d0026bf735eb-ovnkube-config\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.193079 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-modprobe-d\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.193131 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193090 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-kubernetes\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.193131 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193121 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8776ab82-f99c-483e-8001-54e1d88a778d-tmp-dir\") pod \"node-resolver-nqdhp\" (UID: \"8776ab82-f99c-483e-8001-54e1d88a778d\") " pod="openshift-dns/node-resolver-nqdhp" Sep 10 19:15:53.193221 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-sysctl-conf\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.193271 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193227 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5f5j\" (UniqueName: \"kubernetes.io/projected/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-kube-api-access-r5f5j\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.193271 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193253 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-kubelet\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.193368 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193283 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-run-systemd\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.193368 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-run-ovn\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.193368 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193345 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-cni-bin\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.193516 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5510dd56-a971-468d-9909-d0026bf735eb-ovn-node-metrics-cert\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.193516 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-var-lib-kubelet\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.193516 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193432 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff5abec0-8905-4593-bebe-c10577813108-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.193516 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff5abec0-8905-4593-bebe-c10577813108-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.193516 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193498 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-run-netns\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.193774 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193537 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.193774 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5510dd56-a971-468d-9909-d0026bf735eb-env-overrides\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.193774 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193595 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6wp4\" (UniqueName: \"kubernetes.io/projected/8776ab82-f99c-483e-8001-54e1d88a778d-kube-api-access-b6wp4\") pod \"node-resolver-nqdhp\" (UID: \"8776ab82-f99c-483e-8001-54e1d88a778d\") " pod="openshift-dns/node-resolver-nqdhp" Sep 10 19:15:53.193774 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193626 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff5abec0-8905-4593-bebe-c10577813108-system-cni-dir\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.193774 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193656 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-slash\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.193774 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-node-log\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.193774 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-log-socket\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.194097 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193775 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.194097 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5510dd56-a971-468d-9909-d0026bf735eb-ovnkube-script-lib\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.194097 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193838 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-systemd\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.194097 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-tuned\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.194097 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz8jj\" (UniqueName: \"kubernetes.io/projected/ff5abec0-8905-4593-bebe-c10577813108-kube-api-access-rz8jj\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.194097 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193923 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-systemd-units\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.194097 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-etc-openvswitch\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.194097 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.193988 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-cni-netd\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.194097 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.194024 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv7x8\" (UniqueName: \"kubernetes.io/projected/5510dd56-a971-468d-9909-d0026bf735eb-kube-api-access-pv7x8\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.194097 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.194056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-sysctl-d\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.194097 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.194082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-tmp\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.194585 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.194120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8776ab82-f99c-483e-8001-54e1d88a778d-hosts-file\") pod \"node-resolver-nqdhp\" (UID: \"8776ab82-f99c-483e-8001-54e1d88a778d\") " pod="openshift-dns/node-resolver-nqdhp" Sep 10 19:15:53.194585 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.194154 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-var-lib-openvswitch\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.194585 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.194184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-sysconfig\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.194585 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.194213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-lib-modules\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.194585 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.194238 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-host\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.194585 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.194406 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 19:15:53.194897 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.194756 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 19:15:53.195355 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.195336 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 19:15:53.195673 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.195655 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 19:15:53.196018 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.195999 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fs6r4\"" Sep 10 19:15:53.196253 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.196233 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jg9j2\"" Sep 10 19:15:53.196345 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.196284 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dvz6x\"" Sep 10 19:15:53.196591 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.196578 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 19:15:53.196833 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.196815 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 19:15:53.197220 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.197203 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nr8q4\"" Sep 10 19:15:53.197296 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.197273 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 19:15:53.197296 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.197290 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 19:15:53.197821 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.197803 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wnpqz\"" Sep 10 19:15:53.197896 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.197841 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 19:15:53.198609 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.198589 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:15:53.198699 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.198677 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:15:53.200204 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.199917 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 19:15:53.200204 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.200089 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:15:53.200204 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.200157 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:15:53.201686 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.200971 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nxp9t" Sep 10 19:15:53.203018 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.202999 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 19:15:53.203112 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.203097 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5v2sv\"" Sep 10 19:15:53.203887 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.203869 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 19:15:53.204112 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.204093 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:15:53.205665 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.205636 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 19:15:53.215325 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.215286 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:15:53.284807 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.284785 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 19:15:53.294503 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294482 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz8jj\" (UniqueName: \"kubernetes.io/projected/ff5abec0-8905-4593-bebe-c10577813108-kube-api-access-rz8jj\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.294644 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-etc-openvswitch\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.294644 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294538 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv7x8\" (UniqueName: \"kubernetes.io/projected/5510dd56-a971-468d-9909-d0026bf735eb-kube-api-access-pv7x8\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.294644 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs\") pod \"network-metrics-daemon-lr779\" (UID: \"b0837a4a-a1ec-4bda-98f4-d2c9e558894b\") " pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:15:53.294644 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aed52667-53d6-4b29-a7e2-037bb837de84-iptables-alerter-script\") pod \"iptables-alerter-nxp9t\" (UID: \"aed52667-53d6-4b29-a7e2-037bb837de84\") " pod="openshift-network-operator/iptables-alerter-nxp9t" Sep 10 19:15:53.294644 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-sysconfig\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.294644 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294618 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-etc-openvswitch\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.294644 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-lib-modules\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.295013 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294679 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-multus-conf-dir\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.295013 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294710 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqgdd\" (UniqueName: \"kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd\") pod \"network-check-target-sc84z\" (UID: \"1c67355c-4f91-43c1-ac2d-b3c74e5ec44a\") " pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:15:53.295013 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294765 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-lib-modules\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.295013 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294865 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-sysconfig\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.295013 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-run-openvswitch\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.295013 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-run\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.295013 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294970 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-run-openvswitch\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.295013 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.294981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-sys\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.295013 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295004 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0358ef59-3ffa-429f-b980-45ce19f19dab-serviceca\") pod \"node-ca-4b2dk\" (UID: \"0358ef59-3ffa-429f-b980-45ce19f19dab\") " pod="openshift-image-registry/node-ca-4b2dk" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295026 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff5abec0-8905-4593-bebe-c10577813108-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5510dd56-a971-468d-9909-d0026bf735eb-ovnkube-config\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295027 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-sys\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295073 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-run\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295073 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-var-lib-kubelet\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295121 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295147 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-var-lib-cni-bin\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295193 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-etc-kubernetes\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-kubelet\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-var-lib-kubelet\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-registration-dir\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295286 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-kubelet\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a983cad-c60a-47a1-b3b1-964edc3fab7e-cni-binary-copy\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.295458 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-multus-socket-dir-parent\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295605 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5510dd56-a971-468d-9909-d0026bf735eb-ovnkube-config\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295635 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ff5abec0-8905-4593-bebe-c10577813108-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295820 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff5abec0-8905-4593-bebe-c10577813108-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295835 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-var-lib-kubelet\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295869 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-run-netns\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295934 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5510dd56-a971-468d-9909-d0026bf735eb-env-overrides\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295959 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-socket-dir\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295982 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-etc-selinux\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295995 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296007 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-sys-fs\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296028 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-multus-cni-dir\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-node-log\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-log-socket\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-tuned\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.296127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8776ab82-f99c-483e-8001-54e1d88a778d-hosts-file\") pod \"node-resolver-nqdhp\" (UID: \"8776ab82-f99c-483e-8001-54e1d88a778d\") " pod="openshift-dns/node-resolver-nqdhp" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296143 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-system-cni-dir\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296180 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-cnibin\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296201 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-run-netns\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-var-lib-cni-multus\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-systemd-units\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296269 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-cni-netd\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-sysctl-d\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296290 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ff5abec0-8905-4593-bebe-c10577813108-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr2ww\" (UniqueName: \"kubernetes.io/projected/0358ef59-3ffa-429f-b980-45ce19f19dab-kube-api-access-qr2ww\") pod \"node-ca-4b2dk\" (UID: \"0358ef59-3ffa-429f-b980-45ce19f19dab\") " pod="openshift-image-registry/node-ca-4b2dk" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-run-multus-certs\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296356 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8776ab82-f99c-483e-8001-54e1d88a778d-hosts-file\") pod \"node-resolver-nqdhp\" (UID: \"8776ab82-f99c-483e-8001-54e1d88a778d\") " pod="openshift-dns/node-resolver-nqdhp" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-var-lib-openvswitch\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.295959 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-run-netns\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-host\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296395 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-systemd-units\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296413 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f48a3422-593c-4bed-9a93-991b18e65cab-konnectivity-ca\") pod \"konnectivity-agent-bzjwd\" (UID: \"f48a3422-593c-4bed-9a93-991b18e65cab\") " pod="kube-system/konnectivity-agent-bzjwd" Sep 10 19:15:53.296796 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296425 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-node-log\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296365 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5510dd56-a971-468d-9909-d0026bf735eb-env-overrides\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296448 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-device-dir\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckf6s\" (UniqueName: \"kubernetes.io/projected/8a983cad-c60a-47a1-b3b1-964edc3fab7e-kube-api-access-ckf6s\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296491 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-cni-netd\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff5abec0-8905-4593-bebe-c10577813108-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296559 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rv5z\" (UniqueName: \"kubernetes.io/projected/9949612a-3f72-403b-ac12-e520b897f90e-kube-api-access-6rv5z\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296578 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-sysctl-d\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296585 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqw4l\" (UniqueName: \"kubernetes.io/projected/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-kube-api-access-nqw4l\") pod \"network-metrics-daemon-lr779\" (UID: \"b0837a4a-a1ec-4bda-98f4-d2c9e558894b\") " pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqxk5\" (UniqueName: \"kubernetes.io/projected/aed52667-53d6-4b29-a7e2-037bb837de84-kube-api-access-tqxk5\") pod \"iptables-alerter-nxp9t\" (UID: \"aed52667-53d6-4b29-a7e2-037bb837de84\") " pod="openshift-network-operator/iptables-alerter-nxp9t" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296622 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-var-lib-openvswitch\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff5abec0-8905-4593-bebe-c10577813108-cnibin\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296660 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-log-socket\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-modprobe-d\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296698 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-host\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296769 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ff5abec0-8905-4593-bebe-c10577813108-cnibin\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.297617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-kubernetes\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296775 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8776ab82-f99c-483e-8001-54e1d88a778d-tmp-dir\") pod \"node-resolver-nqdhp\" (UID: \"8776ab82-f99c-483e-8001-54e1d88a778d\") " pod="openshift-dns/node-resolver-nqdhp" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296825 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f48a3422-593c-4bed-9a93-991b18e65cab-agent-certs\") pod \"konnectivity-agent-bzjwd\" (UID: \"f48a3422-593c-4bed-9a93-991b18e65cab\") " pod="kube-system/konnectivity-agent-bzjwd" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296847 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-modprobe-d\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aed52667-53d6-4b29-a7e2-037bb837de84-host-slash\") pod \"iptables-alerter-nxp9t\" (UID: \"aed52667-53d6-4b29-a7e2-037bb837de84\") " pod="openshift-network-operator/iptables-alerter-nxp9t" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.296880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-sysctl-conf\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297105 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8776ab82-f99c-483e-8001-54e1d88a778d-tmp-dir\") pod \"node-resolver-nqdhp\" (UID: \"8776ab82-f99c-483e-8001-54e1d88a778d\") " pod="openshift-dns/node-resolver-nqdhp" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5f5j\" (UniqueName: \"kubernetes.io/projected/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-kube-api-access-r5f5j\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-hostroot\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297213 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-run-systemd\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-run-ovn\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297247 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ff5abec0-8905-4593-bebe-c10577813108-cni-binary-copy\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-cni-bin\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297263 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-sysctl-conf\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5510dd56-a971-468d-9909-d0026bf735eb-ovn-node-metrics-cert\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0358ef59-3ffa-429f-b980-45ce19f19dab-host\") pod \"node-ca-4b2dk\" (UID: \"0358ef59-3ffa-429f-b980-45ce19f19dab\") " pod="openshift-image-registry/node-ca-4b2dk" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297449 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff5abec0-8905-4593-bebe-c10577813108-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.298454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297456 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-run-systemd\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297469 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-cni-bin\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6wp4\" (UniqueName: \"kubernetes.io/projected/8776ab82-f99c-483e-8001-54e1d88a778d-kube-api-access-b6wp4\") pod \"node-resolver-nqdhp\" (UID: \"8776ab82-f99c-483e-8001-54e1d88a778d\") " pod="openshift-dns/node-resolver-nqdhp" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297509 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f3cd70c-826e-4955-8532-a9d444835bb8-kubelet-config\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-run-k8s-cni-cncf-io\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297543 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-kubernetes\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297561 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a983cad-c60a-47a1-b3b1-964edc3fab7e-multus-daemon-config\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297599 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-run-ovn\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297629 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff5abec0-8905-4593-bebe-c10577813108-system-cni-dir\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-slash\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297677 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5510dd56-a971-468d-9909-d0026bf735eb-ovnkube-script-lib\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297699 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ff5abec0-8905-4593-bebe-c10577813108-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-systemd\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-tmp\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297911 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f3cd70c-826e-4955-8532-a9d444835bb8-dbus\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297911 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ff5abec0-8905-4593-bebe-c10577813108-system-cni-dir\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.299295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297962 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-systemd\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.300109 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.297997 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-slash\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.300109 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.298056 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5510dd56-a971-468d-9909-d0026bf735eb-host-run-ovn-kubernetes\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.300109 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.298997 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5510dd56-a971-468d-9909-d0026bf735eb-ovnkube-script-lib\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.301943 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.301920 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-etc-tuned\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.302324 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.302295 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-tmp\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.302409 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.302369 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5510dd56-a971-468d-9909-d0026bf735eb-ovn-node-metrics-cert\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.302842 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.302561 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz8jj\" (UniqueName: \"kubernetes.io/projected/ff5abec0-8905-4593-bebe-c10577813108-kube-api-access-rz8jj\") pod \"multus-additional-cni-plugins-xcrxb\" (UID: \"ff5abec0-8905-4593-bebe-c10577813108\") " pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.303644 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.303625 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv7x8\" (UniqueName: \"kubernetes.io/projected/5510dd56-a971-468d-9909-d0026bf735eb-kube-api-access-pv7x8\") pod \"ovnkube-node-75g24\" (UID: \"5510dd56-a971-468d-9909-d0026bf735eb\") " pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.304890 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.304834 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5f5j\" (UniqueName: \"kubernetes.io/projected/24b800ea-3a9b-4bc9-a4e2-880ef10251eb-kube-api-access-r5f5j\") pod \"tuned-q2zsb\" (UID: \"24b800ea-3a9b-4bc9-a4e2-880ef10251eb\") " pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.304890 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.304845 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6wp4\" (UniqueName: \"kubernetes.io/projected/8776ab82-f99c-483e-8001-54e1d88a778d-kube-api-access-b6wp4\") pod \"node-resolver-nqdhp\" (UID: \"8776ab82-f99c-483e-8001-54e1d88a778d\") " pod="openshift-dns/node-resolver-nqdhp" Sep 10 19:15:53.332450 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.332424 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v7rlw" Sep 10 19:15:53.340567 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.340544 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v7rlw" Sep 10 19:15:53.389115 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.389066 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-71.ec2.internal" event={"ID":"d41321aee93f122eed6c1597c05c23f6","Type":"ContainerStarted","Data":"9af7986498b1334f3de2886774a340b8ed17af6956da01d36b31f376a7ffd273"} Sep 10 19:15:53.390432 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.390406 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal" event={"ID":"a79495c71582436fce13ab595ab2b7e7","Type":"ContainerStarted","Data":"148866d357883d9668244ed4b9a40f68bc10e0a65eb2e37c5ce328612cc3f0b2"} Sep 10 19:15:53.398847 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.398822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f48a3422-593c-4bed-9a93-991b18e65cab-konnectivity-ca\") pod \"konnectivity-agent-bzjwd\" (UID: \"f48a3422-593c-4bed-9a93-991b18e65cab\") " pod="kube-system/konnectivity-agent-bzjwd" Sep 10 19:15:53.398943 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.398861 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-device-dir\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.398943 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.398885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckf6s\" (UniqueName: \"kubernetes.io/projected/8a983cad-c60a-47a1-b3b1-964edc3fab7e-kube-api-access-ckf6s\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.398943 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.398912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.398943 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.398935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rv5z\" (UniqueName: \"kubernetes.io/projected/9949612a-3f72-403b-ac12-e520b897f90e-kube-api-access-6rv5z\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.399120 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.398959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqw4l\" (UniqueName: \"kubernetes.io/projected/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-kube-api-access-nqw4l\") pod \"network-metrics-daemon-lr779\" (UID: \"b0837a4a-a1ec-4bda-98f4-d2c9e558894b\") " pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:15:53.399120 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.398983 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqxk5\" (UniqueName: \"kubernetes.io/projected/aed52667-53d6-4b29-a7e2-037bb837de84-kube-api-access-tqxk5\") pod \"iptables-alerter-nxp9t\" (UID: \"aed52667-53d6-4b29-a7e2-037bb837de84\") " pod="openshift-network-operator/iptables-alerter-nxp9t" Sep 10 19:15:53.399120 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399006 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f48a3422-593c-4bed-9a93-991b18e65cab-agent-certs\") pod \"konnectivity-agent-bzjwd\" (UID: \"f48a3422-593c-4bed-9a93-991b18e65cab\") " pod="kube-system/konnectivity-agent-bzjwd" Sep 10 19:15:53.399120 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aed52667-53d6-4b29-a7e2-037bb837de84-host-slash\") pod \"iptables-alerter-nxp9t\" (UID: \"aed52667-53d6-4b29-a7e2-037bb837de84\") " pod="openshift-network-operator/iptables-alerter-nxp9t" Sep 10 19:15:53.399120 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-hostroot\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.399120 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0358ef59-3ffa-429f-b980-45ce19f19dab-host\") pod \"node-ca-4b2dk\" (UID: \"0358ef59-3ffa-429f-b980-45ce19f19dab\") " pod="openshift-image-registry/node-ca-4b2dk" Sep 10 19:15:53.399120 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399101 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f3cd70c-826e-4955-8532-a9d444835bb8-kubelet-config\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-run-k8s-cni-cncf-io\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399146 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a983cad-c60a-47a1-b3b1-964edc3fab7e-multus-daemon-config\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f3cd70c-826e-4955-8532-a9d444835bb8-dbus\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs\") pod \"network-metrics-daemon-lr779\" (UID: \"b0837a4a-a1ec-4bda-98f4-d2c9e558894b\") " pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aed52667-53d6-4b29-a7e2-037bb837de84-iptables-alerter-script\") pod \"iptables-alerter-nxp9t\" (UID: \"aed52667-53d6-4b29-a7e2-037bb837de84\") " pod="openshift-network-operator/iptables-alerter-nxp9t" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399255 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-multus-conf-dir\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgdd\" (UniqueName: \"kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd\") pod \"network-check-target-sc84z\" (UID: \"1c67355c-4f91-43c1-ac2d-b3c74e5ec44a\") " pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0358ef59-3ffa-429f-b980-45ce19f19dab-serviceca\") pod \"node-ca-4b2dk\" (UID: \"0358ef59-3ffa-429f-b980-45ce19f19dab\") " pod="openshift-image-registry/node-ca-4b2dk" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-var-lib-kubelet\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399354 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-var-lib-cni-bin\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-etc-kubernetes\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399412 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0358ef59-3ffa-429f-b980-45ce19f19dab-host\") pod \"node-ca-4b2dk\" (UID: \"0358ef59-3ffa-429f-b980-45ce19f19dab\") " pod="openshift-image-registry/node-ca-4b2dk" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-registration-dir\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a983cad-c60a-47a1-b3b1-964edc3fab7e-cni-binary-copy\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-multus-socket-dir-parent\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.399542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399488 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-socket-dir\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-etc-selinux\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-sys-fs\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399530 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-multus-cni-dir\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-system-cni-dir\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-cnibin\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-run-netns\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399596 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-var-lib-cni-multus\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399618 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr2ww\" (UniqueName: \"kubernetes.io/projected/0358ef59-3ffa-429f-b980-45ce19f19dab-kube-api-access-qr2ww\") pod \"node-ca-4b2dk\" (UID: \"0358ef59-3ffa-429f-b980-45ce19f19dab\") " pod="openshift-image-registry/node-ca-4b2dk" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-run-multus-certs\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-run-multus-certs\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399781 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f3cd70c-826e-4955-8532-a9d444835bb8-kubelet-config\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399823 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-run-k8s-cni-cncf-io\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399931 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f48a3422-593c-4bed-9a93-991b18e65cab-konnectivity-ca\") pod \"konnectivity-agent-bzjwd\" (UID: \"f48a3422-593c-4bed-9a93-991b18e65cab\") " pod="kube-system/konnectivity-agent-bzjwd" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.399999 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-device-dir\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.400269 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.400222 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.400971 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.400350 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8a983cad-c60a-47a1-b3b1-964edc3fab7e-multus-daemon-config\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.400971 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.400518 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f3cd70c-826e-4955-8532-a9d444835bb8-dbus\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:53.400971 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.400605 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:15:53.400971 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.400676 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs podName:b0837a4a-a1ec-4bda-98f4-d2c9e558894b nodeName:}" failed. No retries permitted until 2026-09-10 19:15:53.900655845 +0000 UTC m=+3.031157532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs") pod "network-metrics-daemon-lr779" (UID: "b0837a4a-a1ec-4bda-98f4-d2c9e558894b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:15:53.401282 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.401256 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0358ef59-3ffa-429f-b980-45ce19f19dab-serviceca\") pod \"node-ca-4b2dk\" (UID: \"0358ef59-3ffa-429f-b980-45ce19f19dab\") " pod="openshift-image-registry/node-ca-4b2dk" Sep 10 19:15:53.401418 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.401312 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-var-lib-kubelet\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.401418 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.401386 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:15:53.401534 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.401415 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aed52667-53d6-4b29-a7e2-037bb837de84-host-slash\") pod \"iptables-alerter-nxp9t\" (UID: \"aed52667-53d6-4b29-a7e2-037bb837de84\") " pod="openshift-network-operator/iptables-alerter-nxp9t" Sep 10 19:15:53.401534 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.401429 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret podName:0f3cd70c-826e-4955-8532-a9d444835bb8 nodeName:}" failed. No retries permitted until 2026-09-10 19:15:53.901414992 +0000 UTC m=+3.031916614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret") pod "global-pull-secret-syncer-xdr42" (UID: "0f3cd70c-826e-4955-8532-a9d444835bb8") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:15:53.401534 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.401472 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-var-lib-cni-bin\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.401534 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.401513 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-multus-conf-dir\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.401924 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.401547 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-hostroot\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.402078 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.402046 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/aed52667-53d6-4b29-a7e2-037bb837de84-iptables-alerter-script\") pod \"iptables-alerter-nxp9t\" (UID: \"aed52667-53d6-4b29-a7e2-037bb837de84\") " pod="openshift-network-operator/iptables-alerter-nxp9t" Sep 10 19:15:53.402182 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.402097 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-etc-kubernetes\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.402182 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.402137 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-registration-dir\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.402482 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.402465 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-system-cni-dir\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.402568 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.402509 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-multus-socket-dir-parent\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.402568 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.402531 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a983cad-c60a-47a1-b3b1-964edc3fab7e-cni-binary-copy\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.402667 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.402580 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-socket-dir\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.402667 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.402588 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-cnibin\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.402667 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.402626 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-run-netns\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.402667 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.402647 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-etc-selinux\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.402667 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.402663 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-host-var-lib-cni-multus\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.402919 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.402684 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9949612a-3f72-403b-ac12-e520b897f90e-sys-fs\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.402919 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.402747 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a983cad-c60a-47a1-b3b1-964edc3fab7e-multus-cni-dir\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.403308 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.403293 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f48a3422-593c-4bed-9a93-991b18e65cab-agent-certs\") pod \"konnectivity-agent-bzjwd\" (UID: \"f48a3422-593c-4bed-9a93-991b18e65cab\") " pod="kube-system/konnectivity-agent-bzjwd" Sep 10 19:15:53.408346 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.408257 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:15:53.408346 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.408280 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:15:53.408346 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.408293 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gqgdd for pod openshift-network-diagnostics/network-check-target-sc84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:15:53.408346 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.408342 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd podName:1c67355c-4f91-43c1-ac2d-b3c74e5ec44a nodeName:}" failed. No retries permitted until 2026-09-10 19:15:53.908326994 +0000 UTC m=+3.038828605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gqgdd" (UniqueName: "kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd") pod "network-check-target-sc84z" (UID: "1c67355c-4f91-43c1-ac2d-b3c74e5ec44a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:15:53.408649 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.408628 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqw4l\" (UniqueName: \"kubernetes.io/projected/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-kube-api-access-nqw4l\") pod \"network-metrics-daemon-lr779\" (UID: \"b0837a4a-a1ec-4bda-98f4-d2c9e558894b\") " pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:15:53.409151 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.409123 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqxk5\" (UniqueName: \"kubernetes.io/projected/aed52667-53d6-4b29-a7e2-037bb837de84-kube-api-access-tqxk5\") pod \"iptables-alerter-nxp9t\" (UID: \"aed52667-53d6-4b29-a7e2-037bb837de84\") " pod="openshift-network-operator/iptables-alerter-nxp9t" Sep 10 19:15:53.409346 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.409318 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rv5z\" (UniqueName: \"kubernetes.io/projected/9949612a-3f72-403b-ac12-e520b897f90e-kube-api-access-6rv5z\") pod \"aws-ebs-csi-driver-node-c4sqq\" (UID: \"9949612a-3f72-403b-ac12-e520b897f90e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.411306 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.411267 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckf6s\" (UniqueName: \"kubernetes.io/projected/8a983cad-c60a-47a1-b3b1-964edc3fab7e-kube-api-access-ckf6s\") pod \"multus-rt7fj\" (UID: \"8a983cad-c60a-47a1-b3b1-964edc3fab7e\") " pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.411840 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.411801 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr2ww\" (UniqueName: \"kubernetes.io/projected/0358ef59-3ffa-429f-b980-45ce19f19dab-kube-api-access-qr2ww\") pod \"node-ca-4b2dk\" (UID: \"0358ef59-3ffa-429f-b980-45ce19f19dab\") " pod="openshift-image-registry/node-ca-4b2dk" Sep 10 19:15:53.491749 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.491668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xcrxb" Sep 10 19:15:53.505887 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.505865 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:15:53.514466 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.514448 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" Sep 10 19:15:53.520010 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.519984 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nqdhp" Sep 10 19:15:53.527643 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.527626 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4b2dk" Sep 10 19:15:53.534205 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.534186 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rt7fj" Sep 10 19:15:53.541768 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.541750 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bzjwd" Sep 10 19:15:53.549270 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.549254 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" Sep 10 19:15:53.555764 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.555748 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nxp9t" Sep 10 19:15:53.832737 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.832637 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:15:53.902940 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.902905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs\") pod \"network-metrics-daemon-lr779\" (UID: \"b0837a4a-a1ec-4bda-98f4-d2c9e558894b\") " pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:15:53.903121 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:53.902958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:53.903121 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.903087 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:15:53.903231 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.903147 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret podName:0f3cd70c-826e-4955-8532-a9d444835bb8 nodeName:}" failed. No retries permitted until 2026-09-10 19:15:54.90312768 +0000 UTC m=+4.033629309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret") pod "global-pull-secret-syncer-xdr42" (UID: "0f3cd70c-826e-4955-8532-a9d444835bb8") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:15:53.903550 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.903519 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:15:53.903648 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:53.903577 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs podName:b0837a4a-a1ec-4bda-98f4-d2c9e558894b nodeName:}" failed. No retries permitted until 2026-09-10 19:15:54.903561924 +0000 UTC m=+4.034063540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs") pod "network-metrics-daemon-lr779" (UID: "b0837a4a-a1ec-4bda-98f4-d2c9e558894b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:15:54.003832 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.003798 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgdd\" (UniqueName: \"kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd\") pod \"network-check-target-sc84z\" (UID: \"1c67355c-4f91-43c1-ac2d-b3c74e5ec44a\") " pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:15:54.004020 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:54.003936 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:15:54.004020 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:54.003951 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:15:54.004020 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:54.003960 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gqgdd for pod openshift-network-diagnostics/network-check-target-sc84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:15:54.004020 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:54.004014 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd podName:1c67355c-4f91-43c1-ac2d-b3c74e5ec44a nodeName:}" failed. No retries permitted until 2026-09-10 19:15:55.00399831 +0000 UTC m=+4.134499922 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gqgdd" (UniqueName: "kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd") pod "network-check-target-sc84z" (UID: "1c67355c-4f91-43c1-ac2d-b3c74e5ec44a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:15:54.259582 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:54.259554 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a983cad_c60a_47a1_b3b1_964edc3fab7e.slice/crio-2c3fb276de60b1a8078ac25de3b39fe64fb5a8caaa1be5ffcd758be2bd1a1580 WatchSource:0}: Error finding container 2c3fb276de60b1a8078ac25de3b39fe64fb5a8caaa1be5ffcd758be2bd1a1580: Status 404 returned error can't find the container with id 2c3fb276de60b1a8078ac25de3b39fe64fb5a8caaa1be5ffcd758be2bd1a1580 Sep 10 19:15:54.260678 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:54.260632 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b800ea_3a9b_4bc9_a4e2_880ef10251eb.slice/crio-b38ee6767e8414921772d2973961c40ed98a7c2aa603025a20a22856346fdbff WatchSource:0}: Error finding container b38ee6767e8414921772d2973961c40ed98a7c2aa603025a20a22856346fdbff: Status 404 returned error can't find the container with id b38ee6767e8414921772d2973961c40ed98a7c2aa603025a20a22856346fdbff Sep 10 19:15:54.262652 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:54.262638 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff5abec0_8905_4593_bebe_c10577813108.slice/crio-d6d7e6840fd60ab71fd48f0af891747befd710a4bd525caecef6c36c1fa110c5 WatchSource:0}: Error finding container d6d7e6840fd60ab71fd48f0af891747befd710a4bd525caecef6c36c1fa110c5: Status 404 returned error can't find the container with id d6d7e6840fd60ab71fd48f0af891747befd710a4bd525caecef6c36c1fa110c5 Sep 10 19:15:54.267665 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:54.267637 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5510dd56_a971_468d_9909_d0026bf735eb.slice/crio-41fe410f5a8f396239fe8db6775083987bbafee624cde6d0522c35db903d69d7 WatchSource:0}: Error finding container 41fe410f5a8f396239fe8db6775083987bbafee624cde6d0522c35db903d69d7: Status 404 returned error can't find the container with id 41fe410f5a8f396239fe8db6775083987bbafee624cde6d0522c35db903d69d7 Sep 10 19:15:54.268621 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:54.268593 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed52667_53d6_4b29_a7e2_037bb837de84.slice/crio-494d9508ed29924035f5f955e6d184daed2e2b5d5a09929dd631f26b934195b3 WatchSource:0}: Error finding container 494d9508ed29924035f5f955e6d184daed2e2b5d5a09929dd631f26b934195b3: Status 404 returned error can't find the container with id 494d9508ed29924035f5f955e6d184daed2e2b5d5a09929dd631f26b934195b3 Sep 10 19:15:54.270791 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:54.270141 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0358ef59_3ffa_429f_b980_45ce19f19dab.slice/crio-8cb51a3f3494c424c64d6d25055e79f26cd4ab2aab2ab54f95dcd978a74579d2 WatchSource:0}: Error finding container 8cb51a3f3494c424c64d6d25055e79f26cd4ab2aab2ab54f95dcd978a74579d2: Status 404 returned error can't find the container with id 8cb51a3f3494c424c64d6d25055e79f26cd4ab2aab2ab54f95dcd978a74579d2 Sep 10 19:15:54.272318 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:54.270956 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8776ab82_f99c_483e_8001_54e1d88a778d.slice/crio-9c8a55c8995e42a1510d2058cefb4e24ebef0f62efef840ffe8d8f01f94c1eae WatchSource:0}: Error finding container 9c8a55c8995e42a1510d2058cefb4e24ebef0f62efef840ffe8d8f01f94c1eae: Status 404 returned error can't find the container with id 9c8a55c8995e42a1510d2058cefb4e24ebef0f62efef840ffe8d8f01f94c1eae Sep 10 19:15:54.272318 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:15:54.271917 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9949612a_3f72_403b_ac12_e520b897f90e.slice/crio-00de843824dc79860d123d139329e74c2e9238be4f2080a32a0119b976d37673 WatchSource:0}: Error finding container 00de843824dc79860d123d139329e74c2e9238be4f2080a32a0119b976d37673: Status 404 returned error can't find the container with id 00de843824dc79860d123d139329e74c2e9238be4f2080a32a0119b976d37673 Sep 10 19:15:54.343142 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.343115 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 19:10:53 +0000 UTC" deadline="2028-04-18 00:24:13.035565686 +0000 UTC" Sep 10 19:15:54.343142 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.343141 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14045h8m18.692427298s" Sep 10 19:15:54.393847 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.393822 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" event={"ID":"5510dd56-a971-468d-9909-d0026bf735eb","Type":"ContainerStarted","Data":"41fe410f5a8f396239fe8db6775083987bbafee624cde6d0522c35db903d69d7"} Sep 10 19:15:54.394992 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.394966 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bzjwd" event={"ID":"f48a3422-593c-4bed-9a93-991b18e65cab","Type":"ContainerStarted","Data":"4c538e10166cd4e2e239921dc2bef7c81ced34698f6ca96c4b85da1d113004a5"} Sep 10 19:15:54.399689 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.399299 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-71.ec2.internal" event={"ID":"d41321aee93f122eed6c1597c05c23f6","Type":"ContainerStarted","Data":"73b7a344b6d2e5b209d1078879d257e5252d8de4f992ff847ab01636c981c104"} Sep 10 19:15:54.400522 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.400494 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" event={"ID":"9949612a-3f72-403b-ac12-e520b897f90e","Type":"ContainerStarted","Data":"00de843824dc79860d123d139329e74c2e9238be4f2080a32a0119b976d37673"} Sep 10 19:15:54.401644 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.401609 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4b2dk" event={"ID":"0358ef59-3ffa-429f-b980-45ce19f19dab","Type":"ContainerStarted","Data":"8cb51a3f3494c424c64d6d25055e79f26cd4ab2aab2ab54f95dcd978a74579d2"} Sep 10 19:15:54.402844 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.402821 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcrxb" event={"ID":"ff5abec0-8905-4593-bebe-c10577813108","Type":"ContainerStarted","Data":"d6d7e6840fd60ab71fd48f0af891747befd710a4bd525caecef6c36c1fa110c5"} Sep 10 19:15:54.403858 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.403834 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" event={"ID":"24b800ea-3a9b-4bc9-a4e2-880ef10251eb","Type":"ContainerStarted","Data":"b38ee6767e8414921772d2973961c40ed98a7c2aa603025a20a22856346fdbff"} Sep 10 19:15:54.404977 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.404955 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rt7fj" event={"ID":"8a983cad-c60a-47a1-b3b1-964edc3fab7e","Type":"ContainerStarted","Data":"2c3fb276de60b1a8078ac25de3b39fe64fb5a8caaa1be5ffcd758be2bd1a1580"} Sep 10 19:15:54.405945 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.405925 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nqdhp" event={"ID":"8776ab82-f99c-483e-8001-54e1d88a778d","Type":"ContainerStarted","Data":"9c8a55c8995e42a1510d2058cefb4e24ebef0f62efef840ffe8d8f01f94c1eae"} Sep 10 19:15:54.407045 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.407021 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nxp9t" event={"ID":"aed52667-53d6-4b29-a7e2-037bb837de84","Type":"ContainerStarted","Data":"494d9508ed29924035f5f955e6d184daed2e2b5d5a09929dd631f26b934195b3"} Sep 10 19:15:54.411502 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.411468 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-71.ec2.internal" podStartSLOduration=2.411457081 podStartE2EDuration="2.411457081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:54.411362735 +0000 UTC m=+3.541864365" watchObservedRunningTime="2026-09-10 19:15:54.411457081 +0000 UTC m=+3.541958711" Sep 10 19:15:54.910139 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.909485 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs\") pod \"network-metrics-daemon-lr779\" (UID: \"b0837a4a-a1ec-4bda-98f4-d2c9e558894b\") " pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:15:54.910139 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:54.909561 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:54.910139 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:54.909674 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:15:54.910139 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:54.909671 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:15:54.910139 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:54.909754 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret podName:0f3cd70c-826e-4955-8532-a9d444835bb8 nodeName:}" failed. No retries permitted until 2026-09-10 19:15:56.909715756 +0000 UTC m=+6.040217369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret") pod "global-pull-secret-syncer-xdr42" (UID: "0f3cd70c-826e-4955-8532-a9d444835bb8") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:15:54.910139 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:54.909773 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs podName:b0837a4a-a1ec-4bda-98f4-d2c9e558894b nodeName:}" failed. No retries permitted until 2026-09-10 19:15:56.909763444 +0000 UTC m=+6.040265067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs") pod "network-metrics-daemon-lr779" (UID: "b0837a4a-a1ec-4bda-98f4-d2c9e558894b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:15:55.010292 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:55.010254 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgdd\" (UniqueName: \"kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd\") pod \"network-check-target-sc84z\" (UID: \"1c67355c-4f91-43c1-ac2d-b3c74e5ec44a\") " pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:15:55.010458 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:55.010413 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:15:55.010458 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:55.010431 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:15:55.010458 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:55.010443 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gqgdd for pod openshift-network-diagnostics/network-check-target-sc84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:15:55.010632 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:55.010501 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd podName:1c67355c-4f91-43c1-ac2d-b3c74e5ec44a nodeName:}" failed. No retries permitted until 2026-09-10 19:15:57.010484329 +0000 UTC m=+6.140985952 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gqgdd" (UniqueName: "kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd") pod "network-check-target-sc84z" (UID: "1c67355c-4f91-43c1-ac2d-b3c74e5ec44a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:15:55.189679 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:55.189602 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 19:15:55.343472 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:55.343402 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 19:10:53 +0000 UTC" deadline="2028-03-22 00:14:24.966815489 +0000 UTC" Sep 10 19:15:55.343472 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:55.343463 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13396h58m29.623356759s" Sep 10 19:15:55.387401 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:55.387376 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:55.387551 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:55.387496 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:15:55.387915 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:55.387886 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:15:55.388017 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:55.387997 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:15:55.388076 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:55.388065 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:15:55.388148 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:55.388131 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:15:55.424432 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:55.424385 2578 generic.go:356] "Generic (PLEG): container finished" podID="a79495c71582436fce13ab595ab2b7e7" containerID="f5f479f15f4d51440111088154e9cd9d4dfa6dc06d017a0225143d6793fea485" exitCode=0 Sep 10 19:15:55.425367 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:55.425343 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal" event={"ID":"a79495c71582436fce13ab595ab2b7e7","Type":"ContainerDied","Data":"f5f479f15f4d51440111088154e9cd9d4dfa6dc06d017a0225143d6793fea485"} Sep 10 19:15:56.432746 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:56.432472 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal" event={"ID":"a79495c71582436fce13ab595ab2b7e7","Type":"ContainerStarted","Data":"85e24ada34f967fff0dc719a366f6d5080c38376836e412cd370181d494324cd"} Sep 10 19:15:56.448994 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:56.448943 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-71.ec2.internal" podStartSLOduration=4.448923028 podStartE2EDuration="4.448923028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:15:56.44785146 +0000 UTC m=+5.578353088" watchObservedRunningTime="2026-09-10 19:15:56.448923028 +0000 UTC m=+5.579424661" Sep 10 19:15:56.929798 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:56.926815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs\") pod \"network-metrics-daemon-lr779\" (UID: \"b0837a4a-a1ec-4bda-98f4-d2c9e558894b\") " pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:15:56.929798 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:56.926913 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:56.934838 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:56.930881 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:15:56.934838 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:56.931074 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs podName:b0837a4a-a1ec-4bda-98f4-d2c9e558894b nodeName:}" failed. No retries permitted until 2026-09-10 19:16:00.931050645 +0000 UTC m=+10.061552258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs") pod "network-metrics-daemon-lr779" (UID: "b0837a4a-a1ec-4bda-98f4-d2c9e558894b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:15:56.934838 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:56.931658 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:15:56.934838 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:56.931747 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret podName:0f3cd70c-826e-4955-8532-a9d444835bb8 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:00.931711781 +0000 UTC m=+10.062213395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret") pod "global-pull-secret-syncer-xdr42" (UID: "0f3cd70c-826e-4955-8532-a9d444835bb8") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:15:57.028281 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:57.027638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgdd\" (UniqueName: \"kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd\") pod \"network-check-target-sc84z\" (UID: \"1c67355c-4f91-43c1-ac2d-b3c74e5ec44a\") " pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:15:57.028281 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:57.027866 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:15:57.028281 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:57.027886 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:15:57.028281 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:57.027900 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gqgdd for pod openshift-network-diagnostics/network-check-target-sc84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:15:57.028281 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:57.027958 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd podName:1c67355c-4f91-43c1-ac2d-b3c74e5ec44a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:01.027940359 +0000 UTC m=+10.158441985 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gqgdd" (UniqueName: "kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd") pod "network-check-target-sc84z" (UID: "1c67355c-4f91-43c1-ac2d-b3c74e5ec44a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:15:57.387717 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:57.386848 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:57.387717 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:57.386977 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:15:57.387717 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:57.387048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:15:57.387717 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:57.387171 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:15:57.387717 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:57.387224 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:15:57.387717 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:57.387288 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:15:59.384387 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:59.384355 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:15:59.384857 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:59.384411 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:15:59.384857 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:15:59.384414 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:15:59.384857 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:59.384515 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:15:59.384857 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:59.384551 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:15:59.384857 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:15:59.384668 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:00.961295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:00.960832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs\") pod \"network-metrics-daemon-lr779\" (UID: \"b0837a4a-a1ec-4bda-98f4-d2c9e558894b\") " pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:00.961295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:00.960892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:00.961295 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:00.961015 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:16:00.961295 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:00.961076 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret podName:0f3cd70c-826e-4955-8532-a9d444835bb8 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:08.961058019 +0000 UTC m=+18.091559642 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret") pod "global-pull-secret-syncer-xdr42" (UID: "0f3cd70c-826e-4955-8532-a9d444835bb8") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:16:00.961295 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:00.961225 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:16:00.961295 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:00.961272 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs podName:b0837a4a-a1ec-4bda-98f4-d2c9e558894b nodeName:}" failed. No retries permitted until 2026-09-10 19:16:08.961258614 +0000 UTC m=+18.091760223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs") pod "network-metrics-daemon-lr779" (UID: "b0837a4a-a1ec-4bda-98f4-d2c9e558894b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:16:01.061917 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:01.061883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgdd\" (UniqueName: \"kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd\") pod \"network-check-target-sc84z\" (UID: \"1c67355c-4f91-43c1-ac2d-b3c74e5ec44a\") " pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:01.062095 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:01.062072 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:16:01.062095 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:01.062092 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:16:01.062178 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:01.062105 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gqgdd for pod openshift-network-diagnostics/network-check-target-sc84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:16:01.062178 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:01.062161 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd podName:1c67355c-4f91-43c1-ac2d-b3c74e5ec44a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:09.062143984 +0000 UTC m=+18.192645616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gqgdd" (UniqueName: "kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd") pod "network-check-target-sc84z" (UID: "1c67355c-4f91-43c1-ac2d-b3c74e5ec44a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:16:01.386638 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:01.386558 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:01.386809 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:01.386687 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:16:01.387271 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:01.387248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:01.387382 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:01.387361 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:01.387451 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:01.387425 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:01.388534 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:01.387495 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:16:03.384839 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:03.384808 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:03.385282 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:03.384818 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:03.385282 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:03.384943 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:16:03.385282 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:03.384818 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:03.385282 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:03.385020 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:03.385282 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:03.385101 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:16:05.384878 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:05.384840 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:05.384878 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:05.384864 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:05.385276 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:05.384959 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:05.385276 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:05.385063 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:05.385276 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:05.385160 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:16:05.385276 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:05.385231 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:16:07.385684 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:07.385253 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:07.385684 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:07.385285 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:07.385684 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:07.385268 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:07.385684 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:07.385371 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:16:07.385684 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:07.385435 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:07.385684 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:07.385518 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:16:09.021481 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:09.021443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs\") pod \"network-metrics-daemon-lr779\" (UID: \"b0837a4a-a1ec-4bda-98f4-d2c9e558894b\") " pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:09.021937 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:09.021513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:09.021937 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:09.021597 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:16:09.021937 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:09.021629 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:16:09.021937 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:09.021676 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs podName:b0837a4a-a1ec-4bda-98f4-d2c9e558894b nodeName:}" failed. No retries permitted until 2026-09-10 19:16:25.021655763 +0000 UTC m=+34.152157377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs") pod "network-metrics-daemon-lr779" (UID: "b0837a4a-a1ec-4bda-98f4-d2c9e558894b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:16:09.021937 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:09.021694 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret podName:0f3cd70c-826e-4955-8532-a9d444835bb8 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:25.021687019 +0000 UTC m=+34.152188628 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret") pod "global-pull-secret-syncer-xdr42" (UID: "0f3cd70c-826e-4955-8532-a9d444835bb8") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:16:09.122406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:09.122370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgdd\" (UniqueName: \"kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd\") pod \"network-check-target-sc84z\" (UID: \"1c67355c-4f91-43c1-ac2d-b3c74e5ec44a\") " pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:09.122600 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:09.122577 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:16:09.122660 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:09.122609 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:16:09.122660 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:09.122624 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gqgdd for pod openshift-network-diagnostics/network-check-target-sc84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:16:09.122758 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:09.122690 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd podName:1c67355c-4f91-43c1-ac2d-b3c74e5ec44a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:25.122673582 +0000 UTC m=+34.253175190 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gqgdd" (UniqueName: "kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd") pod "network-check-target-sc84z" (UID: "1c67355c-4f91-43c1-ac2d-b3c74e5ec44a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:16:09.384861 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:09.384771 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:09.385020 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:09.384920 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:16:09.385020 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:09.384770 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:09.385120 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:09.385033 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:09.385120 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:09.384944 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:09.385120 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:09.385113 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:16:11.385447 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:11.385413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:11.385864 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:11.385520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:11.385864 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:11.385556 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:11.385864 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:11.385519 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:16:11.385864 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:11.385642 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:11.385864 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:11.385765 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:16:13.384773 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.384548 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:13.385509 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.384559 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:13.385509 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:13.384809 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:16:13.385509 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.384618 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:13.385509 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:13.384884 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:13.385509 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:13.384963 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:16:13.464266 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.464232 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nqdhp" event={"ID":"8776ab82-f99c-483e-8001-54e1d88a778d","Type":"ContainerStarted","Data":"9d29ba310ef54cca703f655c70c0bc6794878e8160eb2e15990014849feb44b3"} Sep 10 19:16:13.467398 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.467364 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" event={"ID":"5510dd56-a971-468d-9909-d0026bf735eb","Type":"ContainerStarted","Data":"f8eff807d63519295271a8e176d96bfdd303085702715d08edef5f7135ab37b2"} Sep 10 19:16:13.467570 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.467436 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" event={"ID":"5510dd56-a971-468d-9909-d0026bf735eb","Type":"ContainerStarted","Data":"46ad6c4846e74a9ab67f9aa6d6a28ca9c1ccb01d498704ef75294f0efaf75a4c"} Sep 10 19:16:13.467570 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.467488 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" event={"ID":"5510dd56-a971-468d-9909-d0026bf735eb","Type":"ContainerStarted","Data":"ce841f7578e73117bdac9388a4c087d0b5732778ac3dc44a60b591e87317688f"} Sep 10 19:16:13.467570 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.467532 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" event={"ID":"5510dd56-a971-468d-9909-d0026bf735eb","Type":"ContainerStarted","Data":"3bbf1aaee1edc0c4cd636080d6ea2d8bb78975e1d53925682318b9690acea50e"} Sep 10 19:16:13.467712 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.467578 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" event={"ID":"5510dd56-a971-468d-9909-d0026bf735eb","Type":"ContainerStarted","Data":"0d3d410d51c378b1e9dad24f6b23258abafd5a0e3b7e0e287d65deb83163fccf"} Sep 10 19:16:13.467712 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.467626 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" event={"ID":"5510dd56-a971-468d-9909-d0026bf735eb","Type":"ContainerStarted","Data":"dcb05dae28a786015a5c879206448daa2b4b64060dcb03eb5888a0627763004b"} Sep 10 19:16:13.468839 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.468812 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bzjwd" event={"ID":"f48a3422-593c-4bed-9a93-991b18e65cab","Type":"ContainerStarted","Data":"322d00edc57345f5152b23960816997b90e67490b89b84fecdb024809d41add3"} Sep 10 19:16:13.470161 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.470141 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" event={"ID":"9949612a-3f72-403b-ac12-e520b897f90e","Type":"ContainerStarted","Data":"2d1a4a2bc9275ae742f33799ed072e6b811147982e969c60b3612f6bbdcbb69a"} Sep 10 19:16:13.471538 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.471519 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4b2dk" event={"ID":"0358ef59-3ffa-429f-b980-45ce19f19dab","Type":"ContainerStarted","Data":"6e3cc05c5da7cbf7c4d0049d962259e9a818408234f99e092ae0973a95f0187d"} Sep 10 19:16:13.473069 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.473049 2578 generic.go:356] "Generic (PLEG): container finished" podID="ff5abec0-8905-4593-bebe-c10577813108" containerID="b1737013705c9d1a3a414ef64107fa03ebc4ffb868d71486c0012b38e321e808" exitCode=0 Sep 10 19:16:13.473166 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.473112 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcrxb" event={"ID":"ff5abec0-8905-4593-bebe-c10577813108","Type":"ContainerDied","Data":"b1737013705c9d1a3a414ef64107fa03ebc4ffb868d71486c0012b38e321e808"} Sep 10 19:16:13.474635 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.474618 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" event={"ID":"24b800ea-3a9b-4bc9-a4e2-880ef10251eb","Type":"ContainerStarted","Data":"b436875c9ddf6487f111ab8425bed22781b458677f009375b7d747358d9eb617"} Sep 10 19:16:13.476187 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.476168 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rt7fj" event={"ID":"8a983cad-c60a-47a1-b3b1-964edc3fab7e","Type":"ContainerStarted","Data":"72ae98801cd4f4402945835de9ae13dfe907f36ea342bab6c382d04fa51194ff"} Sep 10 19:16:13.478396 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.478363 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nqdhp" podStartSLOduration=4.237110562 podStartE2EDuration="22.478352747s" totalImagesPullingTime="18.241242185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:16:13.477991636 +0000 UTC m=+22.608493267" watchObservedRunningTime="2026-09-10 19:16:13.478352747 +0000 UTC m=+22.608854377" Sep 10 19:16:13.492516 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.492478 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4b2dk" podStartSLOduration=4.272012876 podStartE2EDuration="22.492459045s" totalImagesPullingTime="18.220446169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:16:13.491929724 +0000 UTC m=+22.622431354" watchObservedRunningTime="2026-09-10 19:16:13.492459045 +0000 UTC m=+22.622960675" Sep 10 19:16:13.527400 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.527357 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bzjwd" podStartSLOduration=4.300773684 podStartE2EDuration="22.527345201s" totalImagesPullingTime="18.226571517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:16:13.507705047 +0000 UTC m=+22.638206689" watchObservedRunningTime="2026-09-10 19:16:13.527345201 +0000 UTC m=+22.657846830" Sep 10 19:16:13.545412 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:13.545378 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q2zsb" podStartSLOduration=4.293369481 podStartE2EDuration="22.545365969s" totalImagesPullingTime="18.251996488s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:16:13.545282416 +0000 UTC m=+22.675784048" watchObservedRunningTime="2026-09-10 19:16:13.545365969 +0000 UTC m=+22.675867591" Sep 10 19:16:14.461274 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:14.461129 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 19:16:14.480549 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:14.480522 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nxp9t" event={"ID":"aed52667-53d6-4b29-a7e2-037bb837de84","Type":"ContainerStarted","Data":"50cfec9d68739cff8413470d79e3f2df14c8a36acc4624aa7746492a0602ee0c"} Sep 10 19:16:14.483331 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:14.482521 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" event={"ID":"9949612a-3f72-403b-ac12-e520b897f90e","Type":"ContainerStarted","Data":"52cc2c2cf1bdb97bc656b617767be4e4fdbdf02f1dba7a626c0b9268d38e8aee"} Sep 10 19:16:14.495202 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:14.495162 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rt7fj" podStartSLOduration=5.123068833 podStartE2EDuration="23.495148643s" totalImagesPullingTime="18.37207981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:16:13.562329013 +0000 UTC m=+22.692830644" watchObservedRunningTime="2026-09-10 19:16:14.495148643 +0000 UTC m=+23.625650252" Sep 10 19:16:14.495361 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:14.495342 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nxp9t" podStartSLOduration=5.273971758 podStartE2EDuration="23.495337858s" totalImagesPullingTime="18.2213661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:16:14.495272644 +0000 UTC m=+23.625774273" watchObservedRunningTime="2026-09-10 19:16:14.495337858 +0000 UTC m=+23.625839488" Sep 10 19:16:15.271323 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:15.271234 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T19:16:14.461269861Z","UUID":"e4568b7e-0423-409f-94dd-c062e69085f6","Handler":null,"Name":"","Endpoint":""} Sep 10 19:16:15.272677 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:15.272656 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 19:16:15.272789 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:15.272691 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 19:16:15.385299 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:15.385274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:15.385491 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:15.385400 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:16:15.385816 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:15.385789 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:15.385908 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:15.385873 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:15.386871 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:15.385230 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:15.386994 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:15.386950 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:16:15.489803 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:15.489642 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" event={"ID":"5510dd56-a971-468d-9909-d0026bf735eb","Type":"ContainerStarted","Data":"5b2a7872069b0b13ec7b929f77a152e2ec849ae28f9ab3d5ff7ffe4975a58504"} Sep 10 19:16:15.886774 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:15.886744 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bzjwd" Sep 10 19:16:15.887370 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:15.887349 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bzjwd" Sep 10 19:16:16.494805 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:16.494760 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" event={"ID":"9949612a-3f72-403b-ac12-e520b897f90e","Type":"ContainerStarted","Data":"201cba449b8f84105ad001e695ab1fb3d36e42da3de818ff03031a14d5512c59"} Sep 10 19:16:16.495220 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:16.495164 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bzjwd" Sep 10 19:16:16.495527 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:16.495511 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bzjwd" Sep 10 19:16:16.511889 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:16.511852 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c4sqq" podStartSLOduration=4.411108295 podStartE2EDuration="25.511835815s" totalImagesPullingTime="21.10072752s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:15:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:16:16.511503587 +0000 UTC m=+25.642005218" watchObservedRunningTime="2026-09-10 19:16:16.511835815 +0000 UTC m=+25.642337446" Sep 10 19:16:17.384701 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:17.384224 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:17.384701 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:17.384224 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:17.384701 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:17.384355 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:16:17.384701 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:17.384237 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:17.384701 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:17.384423 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:16:17.384701 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:17.384525 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:18.501987 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:18.501808 2578 generic.go:356] "Generic (PLEG): container finished" podID="ff5abec0-8905-4593-bebe-c10577813108" containerID="9a4529d4a316cdbd9b6aa39492f0bff8a34201f9cf0a522274d9960228713247" exitCode=0 Sep 10 19:16:18.502719 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:18.501888 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcrxb" event={"ID":"ff5abec0-8905-4593-bebe-c10577813108","Type":"ContainerDied","Data":"9a4529d4a316cdbd9b6aa39492f0bff8a34201f9cf0a522274d9960228713247"} Sep 10 19:16:18.505872 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:18.505848 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" event={"ID":"5510dd56-a971-468d-9909-d0026bf735eb","Type":"ContainerStarted","Data":"e15dfe81cf9c7630dd2b1672af859c69c5e826c585cfcf05179dbe1b516c280d"} Sep 10 19:16:18.506247 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:18.506225 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:16:18.506365 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:18.506297 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:16:18.506365 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:18.506346 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:16:18.521617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:18.521598 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:16:18.521793 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:18.521779 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:16:18.550958 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:18.550914 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" podStartSLOduration=9.273404362 podStartE2EDuration="27.550901738s" totalImagesPullingTime="18.277497376s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:16:18.550012803 +0000 UTC m=+27.680514434" watchObservedRunningTime="2026-09-10 19:16:18.550901738 +0000 UTC m=+27.681403369" Sep 10 19:16:19.384702 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:19.384541 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:19.384853 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:19.384827 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:16:19.385246 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:19.385227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:19.385357 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:19.385337 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:19.385418 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:19.385405 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:19.385491 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:19.385474 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:16:19.613811 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:19.613780 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xdr42"] Sep 10 19:16:19.614321 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:19.613873 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:19.614321 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:19.613967 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:16:19.616317 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:19.616296 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lr779"] Sep 10 19:16:19.616435 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:19.616385 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:19.616485 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:19.616475 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:19.628904 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:19.628832 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sc84z"] Sep 10 19:16:19.629039 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:19.628905 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:19.629039 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:19.628988 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:16:20.512041 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:20.512003 2578 generic.go:356] "Generic (PLEG): container finished" podID="ff5abec0-8905-4593-bebe-c10577813108" containerID="6d5539ad57173b9b695af4203a51b55e07136b5c476b17a35e4f8a506479a24e" exitCode=0 Sep 10 19:16:20.512203 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:20.512089 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcrxb" event={"ID":"ff5abec0-8905-4593-bebe-c10577813108","Type":"ContainerDied","Data":"6d5539ad57173b9b695af4203a51b55e07136b5c476b17a35e4f8a506479a24e"} Sep 10 19:16:21.385653 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:21.385618 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:21.386212 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:21.385704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:21.386212 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:21.385767 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:21.386212 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:21.385799 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:16:21.386212 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:21.385845 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:16:21.386212 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:21.385926 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:22.518972 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:22.518700 2578 generic.go:356] "Generic (PLEG): container finished" podID="ff5abec0-8905-4593-bebe-c10577813108" containerID="fcf8fc5cede437a3db2030e6d62960918a85069f15bbcd6806b977c9845ad549" exitCode=0 Sep 10 19:16:22.518972 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:22.518776 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcrxb" event={"ID":"ff5abec0-8905-4593-bebe-c10577813108","Type":"ContainerDied","Data":"fcf8fc5cede437a3db2030e6d62960918a85069f15bbcd6806b977c9845ad549"} Sep 10 19:16:23.386316 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:23.386286 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:23.386505 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:23.386400 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xdr42" podUID="0f3cd70c-826e-4955-8532-a9d444835bb8" Sep 10 19:16:23.386828 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:23.386808 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:23.386926 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:23.386898 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lr779" podUID="b0837a4a-a1ec-4bda-98f4-d2c9e558894b" Sep 10 19:16:23.386980 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:23.386943 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:23.387022 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:23.386988 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-sc84z" podUID="1c67355c-4f91-43c1-ac2d-b3c74e5ec44a" Sep 10 19:16:25.039166 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.039132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs\") pod \"network-metrics-daemon-lr779\" (UID: \"b0837a4a-a1ec-4bda-98f4-d2c9e558894b\") " pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:25.039665 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.039181 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:25.039665 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:25.039289 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 19:16:25.039665 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:25.039294 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:16:25.039665 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:25.039348 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret podName:0f3cd70c-826e-4955-8532-a9d444835bb8 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:57.039332365 +0000 UTC m=+66.169833975 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret") pod "global-pull-secret-syncer-xdr42" (UID: "0f3cd70c-826e-4955-8532-a9d444835bb8") : object "kube-system"/"original-pull-secret" not registered Sep 10 19:16:25.039665 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:25.039363 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs podName:b0837a4a-a1ec-4bda-98f4-d2c9e558894b nodeName:}" failed. No retries permitted until 2026-09-10 19:16:57.039355917 +0000 UTC m=+66.169857525 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs") pod "network-metrics-daemon-lr779" (UID: "b0837a4a-a1ec-4bda-98f4-d2c9e558894b") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 19:16:25.140225 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.140176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgdd\" (UniqueName: \"kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd\") pod \"network-check-target-sc84z\" (UID: \"1c67355c-4f91-43c1-ac2d-b3c74e5ec44a\") " pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:25.140382 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:25.140363 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 19:16:25.140432 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:25.140389 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 19:16:25.140432 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:25.140398 2578 projected.go:196] Error preparing data for projected volume kube-api-access-gqgdd for pod openshift-network-diagnostics/network-check-target-sc84z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:16:25.140509 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:25.140453 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd podName:1c67355c-4f91-43c1-ac2d-b3c74e5ec44a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:57.14043738 +0000 UTC m=+66.270938994 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gqgdd" (UniqueName: "kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd") pod "network-check-target-sc84z" (UID: "1c67355c-4f91-43c1-ac2d-b3c74e5ec44a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 19:16:25.147251 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.147235 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-71.ec2.internal" event="NodeReady" Sep 10 19:16:25.147375 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.147365 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 19:16:25.188815 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.188785 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6hxvz"] Sep 10 19:16:25.191212 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.191193 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:25.191643 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.191619 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jshvk"] Sep 10 19:16:25.193136 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.193118 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jshvk" Sep 10 19:16:25.193384 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.193363 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wks57\"" Sep 10 19:16:25.193487 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.193383 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 19:16:25.193553 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.193487 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 19:16:25.194876 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.194859 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rxhx6\"" Sep 10 19:16:25.195247 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.195228 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 19:16:25.195342 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.195282 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 19:16:25.195487 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.195473 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 19:16:25.203395 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.203355 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6hxvz"] Sep 10 19:16:25.204020 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.204001 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jshvk"] Sep 10 19:16:25.341134 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.341058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/980c5d3f-72f2-4efb-812c-58f2b9ea0d14-cert\") pod \"ingress-canary-jshvk\" (UID: \"980c5d3f-72f2-4efb-812c-58f2b9ea0d14\") " pod="openshift-ingress-canary/ingress-canary-jshvk" Sep 10 19:16:25.341134 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.341106 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97404b03-33d9-40fa-ad37-25daa6ef148d-tmp-dir\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:25.341134 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.341128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvsrt\" (UniqueName: \"kubernetes.io/projected/97404b03-33d9-40fa-ad37-25daa6ef148d-kube-api-access-kvsrt\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:25.341393 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.341211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97404b03-33d9-40fa-ad37-25daa6ef148d-config-volume\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:25.341393 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.341254 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s76xc\" (UniqueName: \"kubernetes.io/projected/980c5d3f-72f2-4efb-812c-58f2b9ea0d14-kube-api-access-s76xc\") pod \"ingress-canary-jshvk\" (UID: \"980c5d3f-72f2-4efb-812c-58f2b9ea0d14\") " pod="openshift-ingress-canary/ingress-canary-jshvk" Sep 10 19:16:25.341393 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.341284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:25.385758 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.384644 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:25.385758 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.384796 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:25.385758 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.384654 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:25.387399 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.387375 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 19:16:25.387399 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.387390 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 19:16:25.387598 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.387437 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tsrfs\"" Sep 10 19:16:25.387654 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.387634 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 19:16:25.387788 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.387682 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q8jbb\"" Sep 10 19:16:25.387933 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.387917 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 19:16:25.442319 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.442295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s76xc\" (UniqueName: \"kubernetes.io/projected/980c5d3f-72f2-4efb-812c-58f2b9ea0d14-kube-api-access-s76xc\") pod \"ingress-canary-jshvk\" (UID: \"980c5d3f-72f2-4efb-812c-58f2b9ea0d14\") " pod="openshift-ingress-canary/ingress-canary-jshvk" Sep 10 19:16:25.442437 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.442335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:25.442437 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.442403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/980c5d3f-72f2-4efb-812c-58f2b9ea0d14-cert\") pod \"ingress-canary-jshvk\" (UID: \"980c5d3f-72f2-4efb-812c-58f2b9ea0d14\") " pod="openshift-ingress-canary/ingress-canary-jshvk" Sep 10 19:16:25.442548 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.442438 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97404b03-33d9-40fa-ad37-25daa6ef148d-tmp-dir\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:25.442548 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:25.442532 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:16:25.442648 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.442563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvsrt\" (UniqueName: \"kubernetes.io/projected/97404b03-33d9-40fa-ad37-25daa6ef148d-kube-api-access-kvsrt\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:25.442648 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:25.442603 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls podName:97404b03-33d9-40fa-ad37-25daa6ef148d nodeName:}" failed. No retries permitted until 2026-09-10 19:16:25.94258257 +0000 UTC m=+35.073084193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls") pod "dns-default-6hxvz" (UID: "97404b03-33d9-40fa-ad37-25daa6ef148d") : secret "dns-default-metrics-tls" not found Sep 10 19:16:25.442805 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.442655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97404b03-33d9-40fa-ad37-25daa6ef148d-config-volume\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:25.442995 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.442963 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97404b03-33d9-40fa-ad37-25daa6ef148d-tmp-dir\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:25.443814 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.443657 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/97404b03-33d9-40fa-ad37-25daa6ef148d-config-volume\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:25.449600 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.449576 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/980c5d3f-72f2-4efb-812c-58f2b9ea0d14-cert\") pod \"ingress-canary-jshvk\" (UID: \"980c5d3f-72f2-4efb-812c-58f2b9ea0d14\") " pod="openshift-ingress-canary/ingress-canary-jshvk" Sep 10 19:16:25.452596 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.452572 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvsrt\" (UniqueName: \"kubernetes.io/projected/97404b03-33d9-40fa-ad37-25daa6ef148d-kube-api-access-kvsrt\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:25.453150 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.453129 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s76xc\" (UniqueName: \"kubernetes.io/projected/980c5d3f-72f2-4efb-812c-58f2b9ea0d14-kube-api-access-s76xc\") pod \"ingress-canary-jshvk\" (UID: \"980c5d3f-72f2-4efb-812c-58f2b9ea0d14\") " pod="openshift-ingress-canary/ingress-canary-jshvk" Sep 10 19:16:25.509664 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.509639 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jshvk" Sep 10 19:16:25.946087 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:25.946049 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:25.946266 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:25.946213 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:16:25.946322 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:25.946297 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls podName:97404b03-33d9-40fa-ad37-25daa6ef148d nodeName:}" failed. No retries permitted until 2026-09-10 19:16:26.946273067 +0000 UTC m=+36.076774699 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls") pod "dns-default-6hxvz" (UID: "97404b03-33d9-40fa-ad37-25daa6ef148d") : secret "dns-default-metrics-tls" not found Sep 10 19:16:26.723604 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:26.723538 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jshvk"] Sep 10 19:16:26.730126 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:16:26.730084 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980c5d3f_72f2_4efb_812c_58f2b9ea0d14.slice/crio-2b58b54d4ce68f9beafccc984da571b264ddbc03634ae206d7148d5b4f4f26a3 WatchSource:0}: Error finding container 2b58b54d4ce68f9beafccc984da571b264ddbc03634ae206d7148d5b4f4f26a3: Status 404 returned error can't find the container with id 2b58b54d4ce68f9beafccc984da571b264ddbc03634ae206d7148d5b4f4f26a3 Sep 10 19:16:26.953798 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:26.953744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:26.953965 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:26.953899 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:16:26.954010 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:26.953968 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls podName:97404b03-33d9-40fa-ad37-25daa6ef148d nodeName:}" failed. No retries permitted until 2026-09-10 19:16:28.953951163 +0000 UTC m=+38.084452776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls") pod "dns-default-6hxvz" (UID: "97404b03-33d9-40fa-ad37-25daa6ef148d") : secret "dns-default-metrics-tls" not found Sep 10 19:16:27.075754 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.075521 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-gfzqc"] Sep 10 19:16:27.077504 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.077489 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:27.079782 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.079757 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 10 19:16:27.079906 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.079830 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 10 19:16:27.079906 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.079877 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-85ppr\"" Sep 10 19:16:27.089424 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.089406 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-gfzqc"] Sep 10 19:16:27.155616 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.155581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/22fced2d-122a-4a5b-b955-0729b7e2ba6d-nginx-conf\") pod \"networking-console-plugin-79977f9844-gfzqc\" (UID: \"22fced2d-122a-4a5b-b955-0729b7e2ba6d\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:27.155802 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.155643 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gfzqc\" (UID: \"22fced2d-122a-4a5b-b955-0729b7e2ba6d\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:27.256436 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.256401 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/22fced2d-122a-4a5b-b955-0729b7e2ba6d-nginx-conf\") pod \"networking-console-plugin-79977f9844-gfzqc\" (UID: \"22fced2d-122a-4a5b-b955-0729b7e2ba6d\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:27.256616 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.256463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gfzqc\" (UID: \"22fced2d-122a-4a5b-b955-0729b7e2ba6d\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:27.256616 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:27.256589 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 19:16:27.256756 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:27.256649 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert podName:22fced2d-122a-4a5b-b955-0729b7e2ba6d nodeName:}" failed. No retries permitted until 2026-09-10 19:16:27.756631574 +0000 UTC m=+36.887133185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gfzqc" (UID: "22fced2d-122a-4a5b-b955-0729b7e2ba6d") : secret "networking-console-plugin-cert" not found Sep 10 19:16:27.257158 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.257135 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/22fced2d-122a-4a5b-b955-0729b7e2ba6d-nginx-conf\") pod \"networking-console-plugin-79977f9844-gfzqc\" (UID: \"22fced2d-122a-4a5b-b955-0729b7e2ba6d\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:27.300980 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.300951 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg"] Sep 10 19:16:27.303001 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.302987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:27.305958 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.305929 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 19:16:27.305958 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.305953 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 10 19:16:27.306144 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.305932 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f8pb4\"" Sep 10 19:16:27.306144 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.306006 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 19:16:27.306144 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.306019 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 10 19:16:27.314842 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.314822 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg"] Sep 10 19:16:27.383768 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.383706 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7c595d49d-6t78r"] Sep 10 19:16:27.385869 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.385849 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-j4bl4"] Sep 10 19:16:27.386028 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.386010 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.387710 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.387686 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.390251 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.390231 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 19:16:27.392705 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.392688 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 10 19:16:27.393199 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.393182 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 10 19:16:27.393456 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.393439 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 19:16:27.393776 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.393759 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 10 19:16:27.393992 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.393977 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cqbf9\"" Sep 10 19:16:27.394208 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.394192 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 10 19:16:27.394437 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.394414 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 10 19:16:27.394700 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.394684 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 10 19:16:27.394981 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.394956 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6k59t\"" Sep 10 19:16:27.395533 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.395510 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 19:16:27.395824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.395797 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 19:16:27.402606 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.402584 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 10 19:16:27.409193 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.409175 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-j4bl4"] Sep 10 19:16:27.410056 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.410036 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7c595d49d-6t78r"] Sep 10 19:16:27.458335 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458306 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de135e89-c44c-41c2-badb-87b930765e4a-tmp\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.458464 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc714527-42ce-4346-bfc8-3e782c031a63-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:27.458464 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458427 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/de135e89-c44c-41c2-badb-87b930765e4a-snapshots\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.458628 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-stats-auth\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.458628 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.458628 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de135e89-c44c-41c2-badb-87b930765e4a-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.458628 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458567 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-default-certificate\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.458628 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458601 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de135e89-c44c-41c2-badb-87b930765e4a-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.458897 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:27.458897 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de135e89-c44c-41c2-badb-87b930765e4a-serving-cert\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.458897 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458696 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5w7\" (UniqueName: \"kubernetes.io/projected/de135e89-c44c-41c2-badb-87b930765e4a-kube-api-access-ft5w7\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.458897 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458744 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-528f6\" (UniqueName: \"kubernetes.io/projected/63f79296-d132-41d4-a902-feeffb16437a-kube-api-access-528f6\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.458897 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458772 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.458897 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.458822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4fhq\" (UniqueName: \"kubernetes.io/projected/bc714527-42ce-4346-bfc8-3e782c031a63-kube-api-access-t4fhq\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:27.537378 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.537348 2578 generic.go:356] "Generic (PLEG): container finished" podID="ff5abec0-8905-4593-bebe-c10577813108" containerID="d4fae0e4644ec3ae5896bbc1fe09bbc92d392f456957fe3b81a041990679504f" exitCode=0 Sep 10 19:16:27.537523 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.537445 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcrxb" event={"ID":"ff5abec0-8905-4593-bebe-c10577813108","Type":"ContainerDied","Data":"d4fae0e4644ec3ae5896bbc1fe09bbc92d392f456957fe3b81a041990679504f"} Sep 10 19:16:27.538813 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.538792 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jshvk" event={"ID":"980c5d3f-72f2-4efb-812c-58f2b9ea0d14","Type":"ContainerStarted","Data":"2b58b54d4ce68f9beafccc984da571b264ddbc03634ae206d7148d5b4f4f26a3"} Sep 10 19:16:27.560431 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560406 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc714527-42ce-4346-bfc8-3e782c031a63-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:27.560546 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/de135e89-c44c-41c2-badb-87b930765e4a-snapshots\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.560546 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560540 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-stats-auth\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.560654 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560564 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.560654 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de135e89-c44c-41c2-badb-87b930765e4a-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.560654 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560615 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-default-certificate\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.560654 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de135e89-c44c-41c2-badb-87b930765e4a-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.560815 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:27.560815 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560712 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de135e89-c44c-41c2-badb-87b930765e4a-serving-cert\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.560815 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560751 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5w7\" (UniqueName: \"kubernetes.io/projected/de135e89-c44c-41c2-badb-87b930765e4a-kube-api-access-ft5w7\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.560815 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560774 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-528f6\" (UniqueName: \"kubernetes.io/projected/63f79296-d132-41d4-a902-feeffb16437a-kube-api-access-528f6\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.560815 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560805 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.561033 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4fhq\" (UniqueName: \"kubernetes.io/projected/bc714527-42ce-4346-bfc8-3e782c031a63-kube-api-access-t4fhq\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:27.561033 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.560866 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de135e89-c44c-41c2-badb-87b930765e4a-tmp\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.561241 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.561220 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de135e89-c44c-41c2-badb-87b930765e4a-tmp\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.561932 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.561909 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc714527-42ce-4346-bfc8-3e782c031a63-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:27.562514 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.562497 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/de135e89-c44c-41c2-badb-87b930765e4a-snapshots\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.562993 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:27.562973 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 19:16:27.563097 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:27.563060 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls podName:bc714527-42ce-4346-bfc8-3e782c031a63 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:28.06304094 +0000 UTC m=+37.193542563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-j69hg" (UID: "bc714527-42ce-4346-bfc8-3e782c031a63") : secret "cluster-monitoring-operator-tls" not found Sep 10 19:16:27.563206 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:27.563190 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 19:16:27.563263 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:27.563236 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs podName:63f79296-d132-41d4-a902-feeffb16437a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:28.063222342 +0000 UTC m=+37.193723957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs") pod "router-default-7c595d49d-6t78r" (UID: "63f79296-d132-41d4-a902-feeffb16437a") : secret "router-metrics-certs-default" not found Sep 10 19:16:27.563476 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:27.563461 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle podName:63f79296-d132-41d4-a902-feeffb16437a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:28.063447277 +0000 UTC m=+37.193948886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle") pod "router-default-7c595d49d-6t78r" (UID: "63f79296-d132-41d4-a902-feeffb16437a") : configmap references non-existent config key: service-ca.crt Sep 10 19:16:27.563978 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.563923 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de135e89-c44c-41c2-badb-87b930765e4a-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.564318 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.564286 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de135e89-c44c-41c2-badb-87b930765e4a-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.566146 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.566123 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/de135e89-c44c-41c2-badb-87b930765e4a-serving-cert\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.566242 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.566170 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-stats-auth\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.569155 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.569116 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-default-certificate\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.574010 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.573983 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-528f6\" (UniqueName: \"kubernetes.io/projected/63f79296-d132-41d4-a902-feeffb16437a-kube-api-access-528f6\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:27.574246 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.574228 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4fhq\" (UniqueName: \"kubernetes.io/projected/bc714527-42ce-4346-bfc8-3e782c031a63-kube-api-access-t4fhq\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:27.576288 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.576262 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5w7\" (UniqueName: \"kubernetes.io/projected/de135e89-c44c-41c2-badb-87b930765e4a-kube-api-access-ft5w7\") pod \"insights-operator-76bcd6ccf5-j4bl4\" (UID: \"de135e89-c44c-41c2-badb-87b930765e4a\") " pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.706568 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.706542 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" Sep 10 19:16:27.764316 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.764093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gfzqc\" (UID: \"22fced2d-122a-4a5b-b955-0729b7e2ba6d\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:27.765362 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:27.764645 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 19:16:27.765362 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:27.764739 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert podName:22fced2d-122a-4a5b-b955-0729b7e2ba6d nodeName:}" failed. No retries permitted until 2026-09-10 19:16:28.764699283 +0000 UTC m=+37.895200951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gfzqc" (UID: "22fced2d-122a-4a5b-b955-0729b7e2ba6d") : secret "networking-console-plugin-cert" not found Sep 10 19:16:27.980523 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:27.980433 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-j4bl4"] Sep 10 19:16:27.983193 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:16:27.983163 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde135e89_c44c_41c2_badb_87b930765e4a.slice/crio-93ca375caa0f0420f7acd720701e48ca9850a8a36590bcb132020179d7b8d51c WatchSource:0}: Error finding container 93ca375caa0f0420f7acd720701e48ca9850a8a36590bcb132020179d7b8d51c: Status 404 returned error can't find the container with id 93ca375caa0f0420f7acd720701e48ca9850a8a36590bcb132020179d7b8d51c Sep 10 19:16:28.068284 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:28.068257 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:28.068446 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:28.068306 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:28.068446 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:28.068349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:28.068446 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:28.068418 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 19:16:28.068637 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:28.068492 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs podName:63f79296-d132-41d4-a902-feeffb16437a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:29.068470974 +0000 UTC m=+38.198972604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs") pod "router-default-7c595d49d-6t78r" (UID: "63f79296-d132-41d4-a902-feeffb16437a") : secret "router-metrics-certs-default" not found Sep 10 19:16:28.068637 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:28.068497 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 19:16:28.068637 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:28.068514 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle podName:63f79296-d132-41d4-a902-feeffb16437a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:29.06850429 +0000 UTC m=+38.199005902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle") pod "router-default-7c595d49d-6t78r" (UID: "63f79296-d132-41d4-a902-feeffb16437a") : configmap references non-existent config key: service-ca.crt Sep 10 19:16:28.068637 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:28.068550 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls podName:bc714527-42ce-4346-bfc8-3e782c031a63 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:29.068533681 +0000 UTC m=+38.199035313 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-j69hg" (UID: "bc714527-42ce-4346-bfc8-3e782c031a63") : secret "cluster-monitoring-operator-tls" not found Sep 10 19:16:28.543017 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:28.542977 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" event={"ID":"de135e89-c44c-41c2-badb-87b930765e4a","Type":"ContainerStarted","Data":"93ca375caa0f0420f7acd720701e48ca9850a8a36590bcb132020179d7b8d51c"} Sep 10 19:16:28.546992 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:28.546962 2578 generic.go:356] "Generic (PLEG): container finished" podID="ff5abec0-8905-4593-bebe-c10577813108" containerID="9da8363ad822db70823e20166543ec0852c62beb4f70fb68efaacd6a92162d19" exitCode=0 Sep 10 19:16:28.547124 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:28.547011 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcrxb" event={"ID":"ff5abec0-8905-4593-bebe-c10577813108","Type":"ContainerDied","Data":"9da8363ad822db70823e20166543ec0852c62beb4f70fb68efaacd6a92162d19"} Sep 10 19:16:28.773921 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:28.773888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gfzqc\" (UID: \"22fced2d-122a-4a5b-b955-0729b7e2ba6d\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:28.774335 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:28.774074 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 19:16:28.774335 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:28.774156 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert podName:22fced2d-122a-4a5b-b955-0729b7e2ba6d nodeName:}" failed. No retries permitted until 2026-09-10 19:16:30.774134325 +0000 UTC m=+39.904635950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gfzqc" (UID: "22fced2d-122a-4a5b-b955-0729b7e2ba6d") : secret "networking-console-plugin-cert" not found Sep 10 19:16:28.975508 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:28.975461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:28.975676 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:28.975609 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:16:28.975676 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:28.975668 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls podName:97404b03-33d9-40fa-ad37-25daa6ef148d nodeName:}" failed. No retries permitted until 2026-09-10 19:16:32.975652253 +0000 UTC m=+42.106153862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls") pod "dns-default-6hxvz" (UID: "97404b03-33d9-40fa-ad37-25daa6ef148d") : secret "dns-default-metrics-tls" not found Sep 10 19:16:29.076152 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:29.076115 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:29.076268 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:29.076175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:29.076268 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:29.076220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:29.076375 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:29.076357 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle podName:63f79296-d132-41d4-a902-feeffb16437a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:31.076338177 +0000 UTC m=+40.206839786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle") pod "router-default-7c595d49d-6t78r" (UID: "63f79296-d132-41d4-a902-feeffb16437a") : configmap references non-existent config key: service-ca.crt Sep 10 19:16:29.076618 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:29.076593 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 19:16:29.076752 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:29.076623 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 19:16:29.076752 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:29.076664 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls podName:bc714527-42ce-4346-bfc8-3e782c031a63 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:31.076646148 +0000 UTC m=+40.207147772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-j69hg" (UID: "bc714527-42ce-4346-bfc8-3e782c031a63") : secret "cluster-monitoring-operator-tls" not found Sep 10 19:16:29.076752 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:29.076681 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs podName:63f79296-d132-41d4-a902-feeffb16437a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:31.076673068 +0000 UTC m=+40.207174676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs") pod "router-default-7c595d49d-6t78r" (UID: "63f79296-d132-41d4-a902-feeffb16437a") : secret "router-metrics-certs-default" not found Sep 10 19:16:29.554765 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:29.554433 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xcrxb" event={"ID":"ff5abec0-8905-4593-bebe-c10577813108","Type":"ContainerStarted","Data":"067d4d48965a0e47d21c00570bd906ffb8981d4fac9f6a0e626ddd03d07ad9d8"} Sep 10 19:16:29.556190 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:29.556162 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jshvk" event={"ID":"980c5d3f-72f2-4efb-812c-58f2b9ea0d14","Type":"ContainerStarted","Data":"d6a56105b7d1e590a7883e9719d8580f7136e8400dc86aa9ed15958c4310d8b0"} Sep 10 19:16:29.576910 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:29.576858 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xcrxb" podStartSLOduration=9.981814565 podStartE2EDuration="38.57684238s" totalImagesPullingTime="28.595027815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:15:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:16:29.575629703 +0000 UTC m=+38.706131333" watchObservedRunningTime="2026-09-10 19:16:29.57684238 +0000 UTC m=+38.707344054" Sep 10 19:16:29.591764 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:29.591641 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jshvk" podStartSLOduration=2.1521289279999998 podStartE2EDuration="4.591621712s" totalImagesPullingTime="2.439492784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:16:29.591359028 +0000 UTC m=+38.721860660" watchObservedRunningTime="2026-09-10 19:16:29.591621712 +0000 UTC m=+38.722123542" Sep 10 19:16:30.793260 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:30.793221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gfzqc\" (UID: \"22fced2d-122a-4a5b-b955-0729b7e2ba6d\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:30.793653 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:30.793322 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 19:16:30.793653 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:30.793372 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert podName:22fced2d-122a-4a5b-b955-0729b7e2ba6d nodeName:}" failed. No retries permitted until 2026-09-10 19:16:34.793359541 +0000 UTC m=+43.923861167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gfzqc" (UID: "22fced2d-122a-4a5b-b955-0729b7e2ba6d") : secret "networking-console-plugin-cert" not found Sep 10 19:16:31.095650 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:31.095625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:31.095767 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:31.095667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:31.095835 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:31.095796 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle podName:63f79296-d132-41d4-a902-feeffb16437a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:35.095781499 +0000 UTC m=+44.226283108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle") pod "router-default-7c595d49d-6t78r" (UID: "63f79296-d132-41d4-a902-feeffb16437a") : configmap references non-existent config key: service-ca.crt Sep 10 19:16:31.095835 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:31.095792 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 19:16:31.095835 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:31.095832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:31.095953 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:31.095859 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls podName:bc714527-42ce-4346-bfc8-3e782c031a63 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:35.09583965 +0000 UTC m=+44.226341277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-j69hg" (UID: "bc714527-42ce-4346-bfc8-3e782c031a63") : secret "cluster-monitoring-operator-tls" not found Sep 10 19:16:31.095953 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:31.095893 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 19:16:31.095953 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:31.095921 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs podName:63f79296-d132-41d4-a902-feeffb16437a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:35.095913389 +0000 UTC m=+44.226414998 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs") pod "router-default-7c595d49d-6t78r" (UID: "63f79296-d132-41d4-a902-feeffb16437a") : secret "router-metrics-certs-default" not found Sep 10 19:16:31.563889 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:31.563852 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" event={"ID":"de135e89-c44c-41c2-badb-87b930765e4a","Type":"ContainerStarted","Data":"ab26479ab0f59b26766deb05867fe58329f412e900001693904801d4c47c4274"} Sep 10 19:16:31.583438 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:31.583382 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" podStartSLOduration=1.486810364 podStartE2EDuration="4.583362509s" totalImagesPullingTime="3.096552145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:16:31.579516541 +0000 UTC m=+40.710018172" watchObservedRunningTime="2026-09-10 19:16:31.583362509 +0000 UTC m=+40.713864164" Sep 10 19:16:32.758079 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:32.758051 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d2dlk"] Sep 10 19:16:32.781768 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:32.781720 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d2dlk"] Sep 10 19:16:32.781891 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:32.781843 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:32.784362 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:32.784338 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q7pt8\"" Sep 10 19:16:32.784496 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:32.784339 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 19:16:32.784496 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:32.784339 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 19:16:32.910156 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:32.910124 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhbwn\" (UniqueName: \"kubernetes.io/projected/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-api-access-fhbwn\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:32.910156 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:32.910159 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-crio-socket\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:32.910373 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:32.910176 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-data-volume\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:32.910373 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:32.910287 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:32.910373 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:32.910311 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:33.011701 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:33.011613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:33.011701 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:33.011675 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:33.011701 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:33.011700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:33.012001 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:33.011720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhbwn\" (UniqueName: \"kubernetes.io/projected/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-api-access-fhbwn\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:33.012001 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:33.011759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-crio-socket\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:33.012001 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:33.011775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-data-volume\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:33.012001 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:33.011816 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:16:33.012001 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:33.011841 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 19:16:33.012001 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:33.011888 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls podName:97404b03-33d9-40fa-ad37-25daa6ef148d nodeName:}" failed. No retries permitted until 2026-09-10 19:16:41.011867811 +0000 UTC m=+50.142369434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls") pod "dns-default-6hxvz" (UID: "97404b03-33d9-40fa-ad37-25daa6ef148d") : secret "dns-default-metrics-tls" not found Sep 10 19:16:33.012001 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:33.011889 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:16:33.012001 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:33.011907 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm podName:ab4d6cda-d5ab-493a-b7aa-0838c7457a55 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:33.511898152 +0000 UTC m=+42.642399771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2dlk" (UID: "ab4d6cda-d5ab-493a-b7aa-0838c7457a55") : configmap "kube-rbac-proxy" not found Sep 10 19:16:33.012001 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:33.011940 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls podName:ab4d6cda-d5ab-493a-b7aa-0838c7457a55 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:33.511924066 +0000 UTC m=+42.642425691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dlk" (UID: "ab4d6cda-d5ab-493a-b7aa-0838c7457a55") : secret "insights-runtime-extractor-tls" not found Sep 10 19:16:33.012001 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:33.011960 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-crio-socket\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:33.012389 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:33.012085 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-data-volume\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:33.028094 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:33.028073 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhbwn\" (UniqueName: \"kubernetes.io/projected/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-api-access-fhbwn\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:33.514973 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:33.514944 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:33.515127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:33.515055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:33.515127 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:33.515104 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:16:33.515127 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:33.515121 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 19:16:33.515273 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:33.515167 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm podName:ab4d6cda-d5ab-493a-b7aa-0838c7457a55 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:34.515155556 +0000 UTC m=+43.645657178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2dlk" (UID: "ab4d6cda-d5ab-493a-b7aa-0838c7457a55") : configmap "kube-rbac-proxy" not found Sep 10 19:16:33.515273 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:33.515180 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls podName:ab4d6cda-d5ab-493a-b7aa-0838c7457a55 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:34.515173044 +0000 UTC m=+43.645674653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dlk" (UID: "ab4d6cda-d5ab-493a-b7aa-0838c7457a55") : secret "insights-runtime-extractor-tls" not found Sep 10 19:16:34.523650 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.523613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:34.524034 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.523660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:34.524034 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:34.523771 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 19:16:34.524034 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:34.523836 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm podName:ab4d6cda-d5ab-493a-b7aa-0838c7457a55 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:36.523820464 +0000 UTC m=+45.654322077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2dlk" (UID: "ab4d6cda-d5ab-493a-b7aa-0838c7457a55") : configmap "kube-rbac-proxy" not found Sep 10 19:16:34.524034 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:34.523837 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:16:34.524034 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:34.523884 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls podName:ab4d6cda-d5ab-493a-b7aa-0838c7457a55 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:36.523872453 +0000 UTC m=+45.654374079 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dlk" (UID: "ab4d6cda-d5ab-493a-b7aa-0838c7457a55") : secret "insights-runtime-extractor-tls" not found Sep 10 19:16:34.697685 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.697657 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nqdhp_8776ab82-f99c-483e-8001-54e1d88a778d/dns-node-resolver/0.log" Sep 10 19:16:34.761702 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.761672 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-2sl8n"] Sep 10 19:16:34.786142 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.786072 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-2sl8n"] Sep 10 19:16:34.786248 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.786155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-2sl8n" Sep 10 19:16:34.789589 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.789554 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 10 19:16:34.789589 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.789569 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 10 19:16:34.790233 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.790216 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 10 19:16:34.790312 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.790296 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7966x\"" Sep 10 19:16:34.790377 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.790312 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 10 19:16:34.825994 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.825967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gfzqc\" (UID: \"22fced2d-122a-4a5b-b955-0729b7e2ba6d\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:34.826120 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:34.826097 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 19:16:34.826159 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:34.826151 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert podName:22fced2d-122a-4a5b-b955-0729b7e2ba6d nodeName:}" failed. No retries permitted until 2026-09-10 19:16:42.826138727 +0000 UTC m=+51.956640351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gfzqc" (UID: "22fced2d-122a-4a5b-b955-0729b7e2ba6d") : secret "networking-console-plugin-cert" not found Sep 10 19:16:34.926971 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.926936 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847c5\" (UniqueName: \"kubernetes.io/projected/f8c540de-0d05-4a66-a59a-aa2bb43fd6dd-kube-api-access-847c5\") pod \"service-ca-75d99c484-2sl8n\" (UID: \"f8c540de-0d05-4a66-a59a-aa2bb43fd6dd\") " pod="openshift-service-ca/service-ca-75d99c484-2sl8n" Sep 10 19:16:34.927075 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.927029 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f8c540de-0d05-4a66-a59a-aa2bb43fd6dd-signing-cabundle\") pod \"service-ca-75d99c484-2sl8n\" (UID: \"f8c540de-0d05-4a66-a59a-aa2bb43fd6dd\") " pod="openshift-service-ca/service-ca-75d99c484-2sl8n" Sep 10 19:16:34.927075 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:34.927056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f8c540de-0d05-4a66-a59a-aa2bb43fd6dd-signing-key\") pod \"service-ca-75d99c484-2sl8n\" (UID: \"f8c540de-0d05-4a66-a59a-aa2bb43fd6dd\") " pod="openshift-service-ca/service-ca-75d99c484-2sl8n" Sep 10 19:16:35.028009 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:35.027979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f8c540de-0d05-4a66-a59a-aa2bb43fd6dd-signing-cabundle\") pod \"service-ca-75d99c484-2sl8n\" (UID: \"f8c540de-0d05-4a66-a59a-aa2bb43fd6dd\") " pod="openshift-service-ca/service-ca-75d99c484-2sl8n" Sep 10 19:16:35.028141 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:35.028027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f8c540de-0d05-4a66-a59a-aa2bb43fd6dd-signing-key\") pod \"service-ca-75d99c484-2sl8n\" (UID: \"f8c540de-0d05-4a66-a59a-aa2bb43fd6dd\") " pod="openshift-service-ca/service-ca-75d99c484-2sl8n" Sep 10 19:16:35.028141 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:35.028093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847c5\" (UniqueName: \"kubernetes.io/projected/f8c540de-0d05-4a66-a59a-aa2bb43fd6dd-kube-api-access-847c5\") pod \"service-ca-75d99c484-2sl8n\" (UID: \"f8c540de-0d05-4a66-a59a-aa2bb43fd6dd\") " pod="openshift-service-ca/service-ca-75d99c484-2sl8n" Sep 10 19:16:35.028594 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:35.028565 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f8c540de-0d05-4a66-a59a-aa2bb43fd6dd-signing-cabundle\") pod \"service-ca-75d99c484-2sl8n\" (UID: \"f8c540de-0d05-4a66-a59a-aa2bb43fd6dd\") " pod="openshift-service-ca/service-ca-75d99c484-2sl8n" Sep 10 19:16:35.030638 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:35.030614 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f8c540de-0d05-4a66-a59a-aa2bb43fd6dd-signing-key\") pod \"service-ca-75d99c484-2sl8n\" (UID: \"f8c540de-0d05-4a66-a59a-aa2bb43fd6dd\") " pod="openshift-service-ca/service-ca-75d99c484-2sl8n" Sep 10 19:16:35.035858 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:35.035837 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-847c5\" (UniqueName: \"kubernetes.io/projected/f8c540de-0d05-4a66-a59a-aa2bb43fd6dd-kube-api-access-847c5\") pod \"service-ca-75d99c484-2sl8n\" (UID: \"f8c540de-0d05-4a66-a59a-aa2bb43fd6dd\") " pod="openshift-service-ca/service-ca-75d99c484-2sl8n" Sep 10 19:16:35.095207 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:35.095124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-2sl8n" Sep 10 19:16:35.128941 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:35.128899 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:35.129046 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:35.128978 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:35.129046 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:35.129022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:35.129046 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:35.129023 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 19:16:35.129220 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:35.129104 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs podName:63f79296-d132-41d4-a902-feeffb16437a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:43.129088216 +0000 UTC m=+52.259589828 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs") pod "router-default-7c595d49d-6t78r" (UID: "63f79296-d132-41d4-a902-feeffb16437a") : secret "router-metrics-certs-default" not found Sep 10 19:16:35.129220 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:35.129104 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 19:16:35.129220 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:35.129155 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle podName:63f79296-d132-41d4-a902-feeffb16437a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:43.129136252 +0000 UTC m=+52.259637890 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle") pod "router-default-7c595d49d-6t78r" (UID: "63f79296-d132-41d4-a902-feeffb16437a") : configmap references non-existent config key: service-ca.crt Sep 10 19:16:35.129220 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:35.129175 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls podName:bc714527-42ce-4346-bfc8-3e782c031a63 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:43.129165875 +0000 UTC m=+52.259667486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-j69hg" (UID: "bc714527-42ce-4346-bfc8-3e782c031a63") : secret "cluster-monitoring-operator-tls" not found Sep 10 19:16:35.375671 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:35.375640 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-2sl8n"] Sep 10 19:16:35.378808 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:16:35.378784 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c540de_0d05_4a66_a59a_aa2bb43fd6dd.slice/crio-114cec6b5131031a3472e78a828ee52acd3c923271605d15d3de04d054774b8e WatchSource:0}: Error finding container 114cec6b5131031a3472e78a828ee52acd3c923271605d15d3de04d054774b8e: Status 404 returned error can't find the container with id 114cec6b5131031a3472e78a828ee52acd3c923271605d15d3de04d054774b8e Sep 10 19:16:35.511103 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:35.511084 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4b2dk_0358ef59-3ffa-429f-b980-45ce19f19dab/node-ca/0.log" Sep 10 19:16:35.573963 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:35.573938 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-2sl8n" event={"ID":"f8c540de-0d05-4a66-a59a-aa2bb43fd6dd","Type":"ContainerStarted","Data":"114cec6b5131031a3472e78a828ee52acd3c923271605d15d3de04d054774b8e"} Sep 10 19:16:36.540664 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:36.540627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:36.540911 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:36.540720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:36.540911 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:36.540800 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:16:36.540911 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:36.540861 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls podName:ab4d6cda-d5ab-493a-b7aa-0838c7457a55 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:40.54084565 +0000 UTC m=+49.671347274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dlk" (UID: "ab4d6cda-d5ab-493a-b7aa-0838c7457a55") : secret "insights-runtime-extractor-tls" not found Sep 10 19:16:36.540911 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:36.540805 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 19:16:36.541052 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:36.540918 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm podName:ab4d6cda-d5ab-493a-b7aa-0838c7457a55 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:40.540906611 +0000 UTC m=+49.671408223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2dlk" (UID: "ab4d6cda-d5ab-493a-b7aa-0838c7457a55") : configmap "kube-rbac-proxy" not found Sep 10 19:16:36.699106 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:36.699077 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jshvk_980c5d3f-72f2-4efb-812c-58f2b9ea0d14/serve-healthcheck-canary/0.log" Sep 10 19:16:38.582939 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:38.582900 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-2sl8n" event={"ID":"f8c540de-0d05-4a66-a59a-aa2bb43fd6dd","Type":"ContainerStarted","Data":"08453dc7ad8c0cc6b5f41ea97fea34b9669588f6be1ea6615817468c83b8dab8"} Sep 10 19:16:38.599889 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:38.599841 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-2sl8n" podStartSLOduration=2.189871777 podStartE2EDuration="4.599824288s" totalImagesPullingTime="2.409952511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:16:38.598610245 +0000 UTC m=+47.729111876" watchObservedRunningTime="2026-09-10 19:16:38.599824288 +0000 UTC m=+47.730325919" Sep 10 19:16:40.578288 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:40.578244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:40.578755 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:40.578367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:40.578755 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:40.578391 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 19:16:40.578755 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:40.578471 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm podName:ab4d6cda-d5ab-493a-b7aa-0838c7457a55 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:48.578449586 +0000 UTC m=+57.708951208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d2dlk" (UID: "ab4d6cda-d5ab-493a-b7aa-0838c7457a55") : configmap "kube-rbac-proxy" not found Sep 10 19:16:40.578755 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:40.578534 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:16:40.578755 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:40.578582 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls podName:ab4d6cda-d5ab-493a-b7aa-0838c7457a55 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:48.578569658 +0000 UTC m=+57.709071267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dlk" (UID: "ab4d6cda-d5ab-493a-b7aa-0838c7457a55") : secret "insights-runtime-extractor-tls" not found Sep 10 19:16:41.083153 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:41.083123 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:41.083324 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:41.083275 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 19:16:41.083368 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:41.083337 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls podName:97404b03-33d9-40fa-ad37-25daa6ef148d nodeName:}" failed. No retries permitted until 2026-09-10 19:16:57.083321634 +0000 UTC m=+66.213823273 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls") pod "dns-default-6hxvz" (UID: "97404b03-33d9-40fa-ad37-25daa6ef148d") : secret "dns-default-metrics-tls" not found Sep 10 19:16:42.894953 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:42.894921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gfzqc\" (UID: \"22fced2d-122a-4a5b-b955-0729b7e2ba6d\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:42.895295 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:42.895071 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 10 19:16:42.895295 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:42.895134 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert podName:22fced2d-122a-4a5b-b955-0729b7e2ba6d nodeName:}" failed. No retries permitted until 2026-09-10 19:16:58.895116313 +0000 UTC m=+68.025617923 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-gfzqc" (UID: "22fced2d-122a-4a5b-b955-0729b7e2ba6d") : secret "networking-console-plugin-cert" not found Sep 10 19:16:43.196650 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:43.196584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:43.196650 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:43.196620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:43.196843 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:43.196654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:43.196843 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:43.196763 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 10 19:16:43.196843 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:43.196763 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 19:16:43.196843 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:43.196775 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle podName:63f79296-d132-41d4-a902-feeffb16437a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:59.19676181 +0000 UTC m=+68.327263431 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle") pod "router-default-7c595d49d-6t78r" (UID: "63f79296-d132-41d4-a902-feeffb16437a") : configmap references non-existent config key: service-ca.crt Sep 10 19:16:43.196973 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:43.196850 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls podName:bc714527-42ce-4346-bfc8-3e782c031a63 nodeName:}" failed. No retries permitted until 2026-09-10 19:16:59.19683028 +0000 UTC m=+68.327331899 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-j69hg" (UID: "bc714527-42ce-4346-bfc8-3e782c031a63") : secret "cluster-monitoring-operator-tls" not found Sep 10 19:16:43.196973 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:43.196908 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs podName:63f79296-d132-41d4-a902-feeffb16437a nodeName:}" failed. No retries permitted until 2026-09-10 19:16:59.196901507 +0000 UTC m=+68.327403116 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs") pod "router-default-7c595d49d-6t78r" (UID: "63f79296-d132-41d4-a902-feeffb16437a") : secret "router-metrics-certs-default" not found Sep 10 19:16:48.642892 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:48.642858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:48.643313 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:48.642948 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:48.643313 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:48.643010 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 19:16:48.643313 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:16:48.643074 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls podName:ab4d6cda-d5ab-493a-b7aa-0838c7457a55 nodeName:}" failed. No retries permitted until 2026-09-10 19:17:04.643059175 +0000 UTC m=+73.773560788 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d2dlk" (UID: "ab4d6cda-d5ab-493a-b7aa-0838c7457a55") : secret "insights-runtime-extractor-tls" not found Sep 10 19:16:48.643454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:48.643435 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:16:50.532220 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:50.532187 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75g24" Sep 10 19:16:57.111205 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.111165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:57.111659 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.111274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:57.111659 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.111300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs\") pod \"network-metrics-daemon-lr779\" (UID: \"b0837a4a-a1ec-4bda-98f4-d2c9e558894b\") " pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:57.113831 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.113810 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 19:16:57.113959 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.113937 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/97404b03-33d9-40fa-ad37-25daa6ef148d-metrics-tls\") pod \"dns-default-6hxvz\" (UID: \"97404b03-33d9-40fa-ad37-25daa6ef148d\") " pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:57.114118 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.114107 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 19:16:57.123794 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.123775 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0837a4a-a1ec-4bda-98f4-d2c9e558894b-metrics-certs\") pod \"network-metrics-daemon-lr779\" (UID: \"b0837a4a-a1ec-4bda-98f4-d2c9e558894b\") " pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:57.123996 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.123974 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f3cd70c-826e-4955-8532-a9d444835bb8-original-pull-secret\") pod \"global-pull-secret-syncer-xdr42\" (UID: \"0f3cd70c-826e-4955-8532-a9d444835bb8\") " pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:57.206234 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.206212 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q8jbb\"" Sep 10 19:16:57.209874 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.209859 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xdr42" Sep 10 19:16:57.212154 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.212138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqgdd\" (UniqueName: \"kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd\") pod \"network-check-target-sc84z\" (UID: \"1c67355c-4f91-43c1-ac2d-b3c74e5ec44a\") " pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:57.214155 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.214139 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 19:16:57.215161 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.215148 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lr779" Sep 10 19:16:57.224578 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.224565 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 19:16:57.235565 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.235546 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqgdd\" (UniqueName: \"kubernetes.io/projected/1c67355c-4f91-43c1-ac2d-b3c74e5ec44a-kube-api-access-gqgdd\") pod \"network-check-target-sc84z\" (UID: \"1c67355c-4f91-43c1-ac2d-b3c74e5ec44a\") " pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:57.305623 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.305458 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wks57\"" Sep 10 19:16:57.312885 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.312534 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6hxvz" Sep 10 19:16:57.499983 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.499955 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tsrfs\"" Sep 10 19:16:57.506030 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.506010 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xdr42"] Sep 10 19:16:57.508028 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.508010 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:16:57.508883 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:16:57.508865 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3cd70c_826e_4955_8532_a9d444835bb8.slice/crio-d12b7cc4d8a439c4ae50920dc3de17d9d63574e34b2332d6285d1a0403c6df55 WatchSource:0}: Error finding container d12b7cc4d8a439c4ae50920dc3de17d9d63574e34b2332d6285d1a0403c6df55: Status 404 returned error can't find the container with id d12b7cc4d8a439c4ae50920dc3de17d9d63574e34b2332d6285d1a0403c6df55 Sep 10 19:16:57.533499 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.533479 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lr779"] Sep 10 19:16:57.536074 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:16:57.536052 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0837a4a_a1ec_4bda_98f4_d2c9e558894b.slice/crio-d63fd3229b3de83edd7aa16d758a6fe57c614f92bb4a87609e07554898621c5f WatchSource:0}: Error finding container d63fd3229b3de83edd7aa16d758a6fe57c614f92bb4a87609e07554898621c5f: Status 404 returned error can't find the container with id d63fd3229b3de83edd7aa16d758a6fe57c614f92bb4a87609e07554898621c5f Sep 10 19:16:57.597904 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.597877 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6hxvz"] Sep 10 19:16:57.605471 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:16:57.605444 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97404b03_33d9_40fa_ad37_25daa6ef148d.slice/crio-a424ea728ce31024b1dec6bb4c15cbd84ce500a9a71b3a4dace5680c625398e2 WatchSource:0}: Error finding container a424ea728ce31024b1dec6bb4c15cbd84ce500a9a71b3a4dace5680c625398e2: Status 404 returned error can't find the container with id a424ea728ce31024b1dec6bb4c15cbd84ce500a9a71b3a4dace5680c625398e2 Sep 10 19:16:57.632280 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.632217 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6hxvz" event={"ID":"97404b03-33d9-40fa-ad37-25daa6ef148d","Type":"ContainerStarted","Data":"a424ea728ce31024b1dec6bb4c15cbd84ce500a9a71b3a4dace5680c625398e2"} Sep 10 19:16:57.633315 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.633286 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xdr42" event={"ID":"0f3cd70c-826e-4955-8532-a9d444835bb8","Type":"ContainerStarted","Data":"d12b7cc4d8a439c4ae50920dc3de17d9d63574e34b2332d6285d1a0403c6df55"} Sep 10 19:16:57.634292 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.634273 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lr779" event={"ID":"b0837a4a-a1ec-4bda-98f4-d2c9e558894b","Type":"ContainerStarted","Data":"d63fd3229b3de83edd7aa16d758a6fe57c614f92bb4a87609e07554898621c5f"} Sep 10 19:16:57.802977 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:57.802939 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-sc84z"] Sep 10 19:16:57.806477 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:16:57.806457 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c67355c_4f91_43c1_ac2d_b3c74e5ec44a.slice/crio-7765395bc48e50f0b66318c65afe35adb78caf78b3b465ac64942c02f011b0e2 WatchSource:0}: Error finding container 7765395bc48e50f0b66318c65afe35adb78caf78b3b465ac64942c02f011b0e2: Status 404 returned error can't find the container with id 7765395bc48e50f0b66318c65afe35adb78caf78b3b465ac64942c02f011b0e2 Sep 10 19:16:58.270247 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.270196 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7"] Sep 10 19:16:58.292457 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.292427 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.295954 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.295934 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 10 19:16:58.296950 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.296698 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 10 19:16:58.296950 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.296698 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 10 19:16:58.296950 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.296702 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 10 19:16:58.296950 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.296702 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 10 19:16:58.297484 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.297467 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 10 19:16:58.297599 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.297560 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 10 19:16:58.310421 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.310397 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7"] Sep 10 19:16:58.322093 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.322069 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6ce7816-417f-4a8a-9c59-fe1899c25a68-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.322203 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.322164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ce7816-417f-4a8a-9c59-fe1899c25a68-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.322203 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.322196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e6ce7816-417f-4a8a-9c59-fe1899c25a68-ca\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.322297 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.322221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65c25\" (UniqueName: \"kubernetes.io/projected/e6ce7816-417f-4a8a-9c59-fe1899c25a68-kube-api-access-65c25\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.322342 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.322307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e6ce7816-417f-4a8a-9c59-fe1899c25a68-hub\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.322374 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.322343 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e6ce7816-417f-4a8a-9c59-fe1899c25a68-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.423635 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.423592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ce7816-417f-4a8a-9c59-fe1899c25a68-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.423635 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.423636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e6ce7816-417f-4a8a-9c59-fe1899c25a68-ca\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.423861 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.423655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65c25\" (UniqueName: \"kubernetes.io/projected/e6ce7816-417f-4a8a-9c59-fe1899c25a68-kube-api-access-65c25\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.423861 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.423676 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e6ce7816-417f-4a8a-9c59-fe1899c25a68-hub\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.423861 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.423692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e6ce7816-417f-4a8a-9c59-fe1899c25a68-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.423861 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.423772 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6ce7816-417f-4a8a-9c59-fe1899c25a68-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.425026 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.424993 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ce7816-417f-4a8a-9c59-fe1899c25a68-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.426920 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.426870 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6ce7816-417f-4a8a-9c59-fe1899c25a68-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.427032 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.427007 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e6ce7816-417f-4a8a-9c59-fe1899c25a68-hub\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.428106 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.428082 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e6ce7816-417f-4a8a-9c59-fe1899c25a68-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.428501 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.428467 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e6ce7816-417f-4a8a-9c59-fe1899c25a68-ca\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.443258 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.443239 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65c25\" (UniqueName: \"kubernetes.io/projected/e6ce7816-417f-4a8a-9c59-fe1899c25a68-kube-api-access-65c25\") pod \"cluster-proxy-proxy-agent-9bf5d549d-n7vz7\" (UID: \"e6ce7816-417f-4a8a-9c59-fe1899c25a68\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.616060 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.615979 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" Sep 10 19:16:58.639529 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.639496 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sc84z" event={"ID":"1c67355c-4f91-43c1-ac2d-b3c74e5ec44a","Type":"ContainerStarted","Data":"7765395bc48e50f0b66318c65afe35adb78caf78b3b465ac64942c02f011b0e2"} Sep 10 19:16:58.926538 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.926504 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7"] Sep 10 19:16:58.928343 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.928311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gfzqc\" (UID: \"22fced2d-122a-4a5b-b955-0729b7e2ba6d\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:58.930036 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:16:58.929983 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6ce7816_417f_4a8a_9c59_fe1899c25a68.slice/crio-a253a9548735f359091bdb40684c722b85d9aa0e56218b3d4b9711b58a42cb35 WatchSource:0}: Error finding container a253a9548735f359091bdb40684c722b85d9aa0e56218b3d4b9711b58a42cb35: Status 404 returned error can't find the container with id a253a9548735f359091bdb40684c722b85d9aa0e56218b3d4b9711b58a42cb35 Sep 10 19:16:58.932623 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:58.932575 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/22fced2d-122a-4a5b-b955-0729b7e2ba6d-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-gfzqc\" (UID: \"22fced2d-122a-4a5b-b955-0729b7e2ba6d\") " pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:59.189127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.188924 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-85ppr\"" Sep 10 19:16:59.197226 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.197019 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" Sep 10 19:16:59.232217 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.231345 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:59.232217 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.231395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:59.232217 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.231428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:59.232217 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.232171 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63f79296-d132-41d4-a902-feeffb16437a-service-ca-bundle\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:59.235218 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.235198 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc714527-42ce-4346-bfc8-3e782c031a63-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-j69hg\" (UID: \"bc714527-42ce-4346-bfc8-3e782c031a63\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:59.237300 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.237258 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63f79296-d132-41d4-a902-feeffb16437a-metrics-certs\") pod \"router-default-7c595d49d-6t78r\" (UID: \"63f79296-d132-41d4-a902-feeffb16437a\") " pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:59.414198 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.414129 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-f8pb4\"" Sep 10 19:16:59.422470 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.422446 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" Sep 10 19:16:59.501462 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.501261 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cqbf9\"" Sep 10 19:16:59.512202 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.509907 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:16:59.523588 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.523472 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-gfzqc"] Sep 10 19:16:59.645246 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.645212 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" event={"ID":"22fced2d-122a-4a5b-b955-0729b7e2ba6d","Type":"ContainerStarted","Data":"c3e9ee29d3cd1a0dd6b2a94d0bf16c4290aebc687c76f162677537780fe63fcd"} Sep 10 19:16:59.647947 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.647919 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lr779" event={"ID":"b0837a4a-a1ec-4bda-98f4-d2c9e558894b","Type":"ContainerStarted","Data":"92c9949a0f8bb798fa72d9a7f8545608bc67abe9f0ee04f79fcc3f9278c12a89"} Sep 10 19:16:59.648066 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.647977 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lr779" event={"ID":"b0837a4a-a1ec-4bda-98f4-d2c9e558894b","Type":"ContainerStarted","Data":"d0960da1deb18ab6132a22327bdf4e56c6fa0053a19261228be6b39cca36c154"} Sep 10 19:16:59.651747 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.651679 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" event={"ID":"e6ce7816-417f-4a8a-9c59-fe1899c25a68","Type":"ContainerStarted","Data":"a253a9548735f359091bdb40684c722b85d9aa0e56218b3d4b9711b58a42cb35"} Sep 10 19:16:59.745366 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.745249 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lr779" podStartSLOduration=67.394055017 podStartE2EDuration="1m8.745224608s" totalImagesPullingTime="1.351169591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:16:59.66568586 +0000 UTC m=+68.796187495" watchObservedRunningTime="2026-09-10 19:16:59.745224608 +0000 UTC m=+68.875726241" Sep 10 19:16:59.745894 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.745867 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg"] Sep 10 19:16:59.751646 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:16:59.750072 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc714527_42ce_4346_bfc8_3e782c031a63.slice/crio-cae1c9a4e73dedcc1af5b57d826cd9546eb77de55282d5d02ac064c17da4b6e5 WatchSource:0}: Error finding container cae1c9a4e73dedcc1af5b57d826cd9546eb77de55282d5d02ac064c17da4b6e5: Status 404 returned error can't find the container with id cae1c9a4e73dedcc1af5b57d826cd9546eb77de55282d5d02ac064c17da4b6e5 Sep 10 19:16:59.830202 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:16:59.829850 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7c595d49d-6t78r"] Sep 10 19:16:59.834692 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:16:59.834662 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f79296_d132_41d4_a902_feeffb16437a.slice/crio-bc46980e89c36ad362d2a0ae7321aeac1469af6a573d83f7544e809bf5b4a189 WatchSource:0}: Error finding container bc46980e89c36ad362d2a0ae7321aeac1469af6a573d83f7544e809bf5b4a189: Status 404 returned error can't find the container with id bc46980e89c36ad362d2a0ae7321aeac1469af6a573d83f7544e809bf5b4a189 Sep 10 19:17:00.661932 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:00.661893 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7c595d49d-6t78r" event={"ID":"63f79296-d132-41d4-a902-feeffb16437a","Type":"ContainerStarted","Data":"e14ec29452f142b30f2a9035036d7e1634fc2e2ef7680e7b72f640f6737fe7d2"} Sep 10 19:17:00.662487 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:00.661960 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7c595d49d-6t78r" event={"ID":"63f79296-d132-41d4-a902-feeffb16437a","Type":"ContainerStarted","Data":"bc46980e89c36ad362d2a0ae7321aeac1469af6a573d83f7544e809bf5b4a189"} Sep 10 19:17:00.665357 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:00.665333 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" event={"ID":"bc714527-42ce-4346-bfc8-3e782c031a63","Type":"ContainerStarted","Data":"cae1c9a4e73dedcc1af5b57d826cd9546eb77de55282d5d02ac064c17da4b6e5"} Sep 10 19:17:00.688899 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:00.688669 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7c595d49d-6t78r" podStartSLOduration=33.68864918 podStartE2EDuration="33.68864918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:00.687284216 +0000 UTC m=+69.817785876" watchObservedRunningTime="2026-09-10 19:17:00.68864918 +0000 UTC m=+69.819150812" Sep 10 19:17:01.510571 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:01.510528 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:17:01.513607 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:01.513572 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:17:01.669451 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:01.669425 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:17:01.670876 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:01.670849 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7c595d49d-6t78r" Sep 10 19:17:04.679046 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:04.679014 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:17:04.681985 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:04.681956 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab4d6cda-d5ab-493a-b7aa-0838c7457a55-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d2dlk\" (UID: \"ab4d6cda-d5ab-493a-b7aa-0838c7457a55\") " pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:17:04.893621 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:04.893586 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q7pt8\"" Sep 10 19:17:04.902480 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:04.902445 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d2dlk" Sep 10 19:17:06.455015 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.454986 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d2dlk"] Sep 10 19:17:06.462648 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:17:06.462624 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4d6cda_d5ab_493a_b7aa_0838c7457a55.slice/crio-320ab451ce0dc89e72489091d594e3e621f0decebf677fdc9866d245a09a568f WatchSource:0}: Error finding container 320ab451ce0dc89e72489091d594e3e621f0decebf677fdc9866d245a09a568f: Status 404 returned error can't find the container with id 320ab451ce0dc89e72489091d594e3e621f0decebf677fdc9866d245a09a568f Sep 10 19:17:06.687238 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.687129 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6hxvz" event={"ID":"97404b03-33d9-40fa-ad37-25daa6ef148d","Type":"ContainerStarted","Data":"95a93fc7448e41cd974214ced57be3b1da37ad96a4096dc8a040fe5c1bce2190"} Sep 10 19:17:06.687238 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.687205 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6hxvz" event={"ID":"97404b03-33d9-40fa-ad37-25daa6ef148d","Type":"ContainerStarted","Data":"73fbefb34fba0d885fac9e42e177861ffadae6cb76ccc824c5fa042a67536ddd"} Sep 10 19:17:06.687238 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.687233 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6hxvz" Sep 10 19:17:06.688852 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.688819 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-sc84z" event={"ID":"1c67355c-4f91-43c1-ac2d-b3c74e5ec44a","Type":"ContainerStarted","Data":"36b487e883f06e954ddbc4a64d4f4d7dbed51b5e7d63378c1623f3b29edb143e"} Sep 10 19:17:06.688986 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.688880 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:17:06.691131 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.691110 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" event={"ID":"e6ce7816-417f-4a8a-9c59-fe1899c25a68","Type":"ContainerStarted","Data":"7012e472a4decd80253d52c9f59e57c6598fdfa3c5c35b1b7b79da2bb03d9f3b"} Sep 10 19:17:06.691238 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.691144 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" event={"ID":"e6ce7816-417f-4a8a-9c59-fe1899c25a68","Type":"ContainerStarted","Data":"9b21f0ebe262803b5a588a7b1bbb74fc29ffb1f003d5b598962f4893f1e20967"} Sep 10 19:17:06.691238 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.691165 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" event={"ID":"e6ce7816-417f-4a8a-9c59-fe1899c25a68","Type":"ContainerStarted","Data":"a321c474290d1c7e730d6f46cf7183373762057582b70621e8f7652ebbd716f2"} Sep 10 19:17:06.693431 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.693403 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xdr42" event={"ID":"0f3cd70c-826e-4955-8532-a9d444835bb8","Type":"ContainerStarted","Data":"e54c67ce6c13cd2dc2c7b233219c8fb61daadb067a7218b60bc16ffb968db30e"} Sep 10 19:17:06.695894 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.695865 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" event={"ID":"22fced2d-122a-4a5b-b955-0729b7e2ba6d","Type":"ContainerStarted","Data":"8bdf1c64253dc5c65576a5037fb802fad14d515c6a86990c08ee518ca5a314d7"} Sep 10 19:17:06.697674 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.697651 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" event={"ID":"bc714527-42ce-4346-bfc8-3e782c031a63","Type":"ContainerStarted","Data":"d79b6eccf280678ebd3e61610bfcc9dc57b0773d1a597a08fca31ff1c3bbd950"} Sep 10 19:17:06.699812 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.699784 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2dlk" event={"ID":"ab4d6cda-d5ab-493a-b7aa-0838c7457a55","Type":"ContainerStarted","Data":"9a4bd1b09c9f0c561766a89941690024d6120c85194e78244f08e6d5926b536b"} Sep 10 19:17:06.699909 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.699843 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2dlk" event={"ID":"ab4d6cda-d5ab-493a-b7aa-0838c7457a55","Type":"ContainerStarted","Data":"320ab451ce0dc89e72489091d594e3e621f0decebf677fdc9866d245a09a568f"} Sep 10 19:17:06.707452 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.707408 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6hxvz" podStartSLOduration=33.141571255 podStartE2EDuration="41.707396229s" totalImagesPullingTime="8.565824974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:06.706437339 +0000 UTC m=+75.836938974" watchObservedRunningTime="2026-09-10 19:17:06.707396229 +0000 UTC m=+75.837897859" Sep 10 19:17:06.744265 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.744216 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-xdr42" podStartSLOduration=67.59676042 podStartE2EDuration="1m15.74419777s" totalImagesPullingTime="8.14743735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:06.724568832 +0000 UTC m=+75.855070464" watchObservedRunningTime="2026-09-10 19:17:06.74419777 +0000 UTC m=+75.874699403" Sep 10 19:17:06.744782 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.744754 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bf5d549d-n7vz7" podStartSLOduration=1.4823823219999999 podStartE2EDuration="8.744713313s" totalImagesPullingTime="7.262330991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:06.743327216 +0000 UTC m=+75.873828847" watchObservedRunningTime="2026-09-10 19:17:06.744713313 +0000 UTC m=+75.875214987" Sep 10 19:17:06.761310 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.761267 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-sc84z" podStartSLOduration=67.396581035 podStartE2EDuration="1m15.761248956s" totalImagesPullingTime="8.364667921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:06.759973453 +0000 UTC m=+75.890475097" watchObservedRunningTime="2026-09-10 19:17:06.761248956 +0000 UTC m=+75.891750590" Sep 10 19:17:06.780417 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.780376 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-gfzqc" podStartSLOduration=33.138021599 podStartE2EDuration="39.780363262s" totalImagesPullingTime="6.642341663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:06.779556708 +0000 UTC m=+75.910058342" watchObservedRunningTime="2026-09-10 19:17:06.780363262 +0000 UTC m=+75.910864957" Sep 10 19:17:06.805659 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:06.805607 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-j69hg" podStartSLOduration=33.380117366 podStartE2EDuration="39.805593881s" totalImagesPullingTime="6.425476515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:06.805336843 +0000 UTC m=+75.935838475" watchObservedRunningTime="2026-09-10 19:17:06.805593881 +0000 UTC m=+75.936095510" Sep 10 19:17:07.706204 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:07.706173 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2dlk" event={"ID":"ab4d6cda-d5ab-493a-b7aa-0838c7457a55","Type":"ContainerStarted","Data":"d4f7119fe0d49aa07640542da89e043378cd9d71dbd3da9e90d07c8ec9800a9e"} Sep 10 19:17:09.717288 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:09.717248 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d2dlk" event={"ID":"ab4d6cda-d5ab-493a-b7aa-0838c7457a55","Type":"ContainerStarted","Data":"e53690ce59d0b8d3e2846f78039491a60d603792567ca77e363c54da8c33ec11"} Sep 10 19:17:09.737932 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:09.737879 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d2dlk" podStartSLOduration=35.264112155 podStartE2EDuration="37.737862745s" totalImagesPullingTime="2.47375059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:16:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:09.736960361 +0000 UTC m=+78.867461993" watchObservedRunningTime="2026-09-10 19:17:09.737862745 +0000 UTC m=+78.868364399" Sep 10 19:17:16.709026 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:16.708998 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6hxvz" Sep 10 19:17:25.208977 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.208944 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9492w"] Sep 10 19:17:25.212131 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.212115 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.214262 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.214239 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 19:17:25.215205 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.215186 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 19:17:25.215307 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.215217 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gfqql\"" Sep 10 19:17:25.215307 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.215240 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 19:17:25.215307 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.215185 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 19:17:25.321041 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.321012 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdffp\" (UniqueName: \"kubernetes.io/projected/98770593-3fdd-4367-9d4b-8859a01db7b3-kube-api-access-cdffp\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.321172 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.321054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-textfile\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.321172 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.321081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.321295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.321184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98770593-3fdd-4367-9d4b-8859a01db7b3-metrics-client-ca\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.321295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.321224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-wtmp\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.321295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.321258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-tls\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.321295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.321291 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98770593-3fdd-4367-9d4b-8859a01db7b3-sys\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.321438 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.321318 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/98770593-3fdd-4367-9d4b-8859a01db7b3-root\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.321438 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.321344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.422236 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.422204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-wtmp\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.422391 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.422248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-tls\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.422391 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.422278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98770593-3fdd-4367-9d4b-8859a01db7b3-sys\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.422391 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.422303 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/98770593-3fdd-4367-9d4b-8859a01db7b3-root\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.422391 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.422327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.422391 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.422361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdffp\" (UniqueName: \"kubernetes.io/projected/98770593-3fdd-4367-9d4b-8859a01db7b3-kube-api-access-cdffp\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.422391 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.422385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-textfile\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.422695 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.422398 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-wtmp\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.422695 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.422410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.422695 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.422428 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/98770593-3fdd-4367-9d4b-8859a01db7b3-root\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.422695 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.422516 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98770593-3fdd-4367-9d4b-8859a01db7b3-metrics-client-ca\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.422695 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.422520 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/98770593-3fdd-4367-9d4b-8859a01db7b3-sys\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.423285 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.423117 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/98770593-3fdd-4367-9d4b-8859a01db7b3-metrics-client-ca\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.423285 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.423209 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.423285 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.423214 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-textfile\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.425239 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.425218 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-tls\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.425518 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.425499 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/98770593-3fdd-4367-9d4b-8859a01db7b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.431016 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.430993 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdffp\" (UniqueName: \"kubernetes.io/projected/98770593-3fdd-4367-9d4b-8859a01db7b3-kube-api-access-cdffp\") pod \"node-exporter-9492w\" (UID: \"98770593-3fdd-4367-9d4b-8859a01db7b3\") " pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.523672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.523612 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9492w" Sep 10 19:17:25.532895 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:17:25.532859 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98770593_3fdd_4367_9d4b_8859a01db7b3.slice/crio-0a8ca11a5733f43ad4eb343ab96302fddd1bd922ebf2f00ddd277eb2c0f19e68 WatchSource:0}: Error finding container 0a8ca11a5733f43ad4eb343ab96302fddd1bd922ebf2f00ddd277eb2c0f19e68: Status 404 returned error can't find the container with id 0a8ca11a5733f43ad4eb343ab96302fddd1bd922ebf2f00ddd277eb2c0f19e68 Sep 10 19:17:25.769264 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:25.769231 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9492w" event={"ID":"98770593-3fdd-4367-9d4b-8859a01db7b3","Type":"ContainerStarted","Data":"0a8ca11a5733f43ad4eb343ab96302fddd1bd922ebf2f00ddd277eb2c0f19e68"} Sep 10 19:17:27.777999 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:27.777959 2578 generic.go:356] "Generic (PLEG): container finished" podID="98770593-3fdd-4367-9d4b-8859a01db7b3" containerID="f0518339b0b0c56d33f401744c2eba6087e3e3fe1a29237fc717d47d43890a57" exitCode=0 Sep 10 19:17:27.778429 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:27.778050 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9492w" event={"ID":"98770593-3fdd-4367-9d4b-8859a01db7b3","Type":"ContainerDied","Data":"f0518339b0b0c56d33f401744c2eba6087e3e3fe1a29237fc717d47d43890a57"} Sep 10 19:17:28.783070 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:28.783031 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9492w" event={"ID":"98770593-3fdd-4367-9d4b-8859a01db7b3","Type":"ContainerStarted","Data":"5aba0a8cb8089381355ebc9a8128bd0e8a017ebab784b7fde654645f44741c14"} Sep 10 19:17:28.783444 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:28.783084 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9492w" event={"ID":"98770593-3fdd-4367-9d4b-8859a01db7b3","Type":"ContainerStarted","Data":"9c5702b75c22c49eae671e99b83a2a0f2f5b1656c4ddd22058db5464914ad154"} Sep 10 19:17:28.802925 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:28.802883 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9492w" podStartSLOduration=2.51024598 podStartE2EDuration="3.802869017s" totalImagesPullingTime="1.221625226s" totalInitContainerRuntime="70.997811ms" isStatefulPod=true podCreationTimestamp="2026-09-10 19:17:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:28.80218336 +0000 UTC m=+97.932684991" watchObservedRunningTime="2026-09-10 19:17:28.802869017 +0000 UTC m=+97.933370647" Sep 10 19:17:30.465289 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.465255 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-77bb685b6d-n57b9"] Sep 10 19:17:30.467482 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.467464 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.470782 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.470760 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 10 19:17:30.472779 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.472759 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 10 19:17:30.473021 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.473000 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-w7kd9\"" Sep 10 19:17:30.474153 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.474134 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 10 19:17:30.474466 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.474447 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 10 19:17:30.474984 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.474960 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 10 19:17:30.479570 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.479553 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 10 19:17:30.489001 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.488978 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77bb685b6d-n57b9"] Sep 10 19:17:30.560319 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.560293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd4f2784-48b4-4050-b12a-b882ce154c73-metrics-client-ca\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.560420 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.560324 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfmhs\" (UniqueName: \"kubernetes.io/projected/fd4f2784-48b4-4050-b12a-b882ce154c73-kube-api-access-vfmhs\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.560420 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.560358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fd4f2784-48b4-4050-b12a-b882ce154c73-telemeter-client-tls\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.560420 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.560407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd4f2784-48b4-4050-b12a-b882ce154c73-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.560546 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.560445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fd4f2784-48b4-4050-b12a-b882ce154c73-federate-client-tls\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.560546 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.560468 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd4f2784-48b4-4050-b12a-b882ce154c73-serving-certs-ca-bundle\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.560546 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.560482 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fd4f2784-48b4-4050-b12a-b882ce154c73-secret-telemeter-client\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.560636 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.560556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd4f2784-48b4-4050-b12a-b882ce154c73-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.661774 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.661718 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fd4f2784-48b4-4050-b12a-b882ce154c73-federate-client-tls\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.661908 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.661781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd4f2784-48b4-4050-b12a-b882ce154c73-serving-certs-ca-bundle\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.661908 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.661797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fd4f2784-48b4-4050-b12a-b882ce154c73-secret-telemeter-client\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.661908 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.661850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd4f2784-48b4-4050-b12a-b882ce154c73-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.661908 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.661877 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd4f2784-48b4-4050-b12a-b882ce154c73-metrics-client-ca\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.661908 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.661902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfmhs\" (UniqueName: \"kubernetes.io/projected/fd4f2784-48b4-4050-b12a-b882ce154c73-kube-api-access-vfmhs\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.662142 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.662014 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fd4f2784-48b4-4050-b12a-b882ce154c73-telemeter-client-tls\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.662142 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.662045 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd4f2784-48b4-4050-b12a-b882ce154c73-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.662629 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.662602 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd4f2784-48b4-4050-b12a-b882ce154c73-metrics-client-ca\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.662712 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.662612 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd4f2784-48b4-4050-b12a-b882ce154c73-serving-certs-ca-bundle\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.663054 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.662925 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fd4f2784-48b4-4050-b12a-b882ce154c73-telemeter-trusted-ca-bundle\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.664512 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.664486 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fd4f2784-48b4-4050-b12a-b882ce154c73-federate-client-tls\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.664765 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.664742 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fd4f2784-48b4-4050-b12a-b882ce154c73-telemeter-client-tls\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.664812 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.664799 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fd4f2784-48b4-4050-b12a-b882ce154c73-secret-telemeter-client\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.664846 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.664815 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd4f2784-48b4-4050-b12a-b882ce154c73-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.671860 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.671838 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfmhs\" (UniqueName: \"kubernetes.io/projected/fd4f2784-48b4-4050-b12a-b882ce154c73-kube-api-access-vfmhs\") pod \"telemeter-client-77bb685b6d-n57b9\" (UID: \"fd4f2784-48b4-4050-b12a-b882ce154c73\") " pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:30.776833 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:30.776765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" Sep 10 19:17:31.049807 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:31.049716 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-77bb685b6d-n57b9"] Sep 10 19:17:31.052624 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:17:31.052587 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd4f2784_48b4_4050_b12a_b882ce154c73.slice/crio-91e1fb55d2355cbe9a34b39cb79f3b18f1e48437295270780ad702d674a39390 WatchSource:0}: Error finding container 91e1fb55d2355cbe9a34b39cb79f3b18f1e48437295270780ad702d674a39390: Status 404 returned error can't find the container with id 91e1fb55d2355cbe9a34b39cb79f3b18f1e48437295270780ad702d674a39390 Sep 10 19:17:31.794884 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:31.794846 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" event={"ID":"fd4f2784-48b4-4050-b12a-b882ce154c73","Type":"ContainerStarted","Data":"91e1fb55d2355cbe9a34b39cb79f3b18f1e48437295270780ad702d674a39390"} Sep 10 19:17:33.804113 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:33.804032 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" event={"ID":"fd4f2784-48b4-4050-b12a-b882ce154c73","Type":"ContainerStarted","Data":"4fa6775a3a247b7fc00fab3b366b3c95f5f8f8dafe50ba4a4500983a855d1302"} Sep 10 19:17:35.813179 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:35.813139 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" event={"ID":"fd4f2784-48b4-4050-b12a-b882ce154c73","Type":"ContainerStarted","Data":"aa0a9edd95ddd0d212a3e1e5ceb5acb8dc1236e511399733e0f33174cd740a40"} Sep 10 19:17:35.813661 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:35.813203 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" event={"ID":"fd4f2784-48b4-4050-b12a-b882ce154c73","Type":"ContainerStarted","Data":"0ae8bda2562c3e7f5602faf7912790f4974a26aebe01e7c6e32d4aa1b0addfc8"} Sep 10 19:17:35.838211 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:35.838166 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-77bb685b6d-n57b9" podStartSLOduration=1.789685363 podStartE2EDuration="5.83814607s" totalImagesPullingTime="4.048460707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:17:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:17:35.836454205 +0000 UTC m=+104.966955837" watchObservedRunningTime="2026-09-10 19:17:35.83814607 +0000 UTC m=+104.968647722" Sep 10 19:17:37.709087 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:37.709057 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-sc84z" Sep 10 19:17:42.838406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:42.838375 2578 generic.go:356] "Generic (PLEG): container finished" podID="de135e89-c44c-41c2-badb-87b930765e4a" containerID="ab26479ab0f59b26766deb05867fe58329f412e900001693904801d4c47c4274" exitCode=0 Sep 10 19:17:42.838782 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:42.838453 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" event={"ID":"de135e89-c44c-41c2-badb-87b930765e4a","Type":"ContainerDied","Data":"ab26479ab0f59b26766deb05867fe58329f412e900001693904801d4c47c4274"} Sep 10 19:17:42.838832 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:42.838817 2578 scope.go:117] "RemoveContainer" containerID="ab26479ab0f59b26766deb05867fe58329f412e900001693904801d4c47c4274" Sep 10 19:17:43.699161 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:43.699135 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c595d49d-6t78r_63f79296-d132-41d4-a902-feeffb16437a/router/0.log" Sep 10 19:17:43.720233 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:43.720206 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jshvk_980c5d3f-72f2-4efb-812c-58f2b9ea0d14/serve-healthcheck-canary/0.log" Sep 10 19:17:43.843900 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:17:43.843874 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-j4bl4" event={"ID":"de135e89-c44c-41c2-badb-87b930765e4a","Type":"ContainerStarted","Data":"2f2a95e474a163fa22aa9833206c2ec5faf4ea15232febdbb40134d0b5352bc3"} Sep 10 19:19:53.485987 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:53.485958 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl"] Sep 10 19:19:53.487696 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:53.487673 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl" Sep 10 19:19:53.490597 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:53.490578 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 10 19:19:53.490777 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:53.490758 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:19:53.502542 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:53.502525 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9cjvx\"" Sep 10 19:19:53.516538 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:53.516511 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl"] Sep 10 19:19:53.574415 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:53.574391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48b8j\" (UniqueName: \"kubernetes.io/projected/0ffb7da0-bdf2-4cf1-8fd9-82372e27409f-kube-api-access-48b8j\") pod \"cert-manager-operator-controller-manager-55c444bf86-wb6hl\" (UID: \"0ffb7da0-bdf2-4cf1-8fd9-82372e27409f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl" Sep 10 19:19:53.574511 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:53.574446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ffb7da0-bdf2-4cf1-8fd9-82372e27409f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wb6hl\" (UID: \"0ffb7da0-bdf2-4cf1-8fd9-82372e27409f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl" Sep 10 19:19:53.675501 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:53.675477 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ffb7da0-bdf2-4cf1-8fd9-82372e27409f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wb6hl\" (UID: \"0ffb7da0-bdf2-4cf1-8fd9-82372e27409f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl" Sep 10 19:19:53.675639 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:53.675512 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48b8j\" (UniqueName: \"kubernetes.io/projected/0ffb7da0-bdf2-4cf1-8fd9-82372e27409f-kube-api-access-48b8j\") pod \"cert-manager-operator-controller-manager-55c444bf86-wb6hl\" (UID: \"0ffb7da0-bdf2-4cf1-8fd9-82372e27409f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl" Sep 10 19:19:53.675904 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:53.675883 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ffb7da0-bdf2-4cf1-8fd9-82372e27409f-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-wb6hl\" (UID: \"0ffb7da0-bdf2-4cf1-8fd9-82372e27409f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl" Sep 10 19:19:53.684058 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:53.684032 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48b8j\" (UniqueName: \"kubernetes.io/projected/0ffb7da0-bdf2-4cf1-8fd9-82372e27409f-kube-api-access-48b8j\") pod \"cert-manager-operator-controller-manager-55c444bf86-wb6hl\" (UID: \"0ffb7da0-bdf2-4cf1-8fd9-82372e27409f\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl" Sep 10 19:19:53.796023 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:53.795950 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl" Sep 10 19:19:54.071235 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:54.071170 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl"] Sep 10 19:19:54.076182 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:19:54.076156 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ffb7da0_bdf2_4cf1_8fd9_82372e27409f.slice/crio-41455fbc567d03466ba9710fd76bcbd037506aecc65f08e3dff3c33df8b3cd0a WatchSource:0}: Error finding container 41455fbc567d03466ba9710fd76bcbd037506aecc65f08e3dff3c33df8b3cd0a: Status 404 returned error can't find the container with id 41455fbc567d03466ba9710fd76bcbd037506aecc65f08e3dff3c33df8b3cd0a Sep 10 19:19:54.295701 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:54.295671 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl" event={"ID":"0ffb7da0-bdf2-4cf1-8fd9-82372e27409f","Type":"ContainerStarted","Data":"41455fbc567d03466ba9710fd76bcbd037506aecc65f08e3dff3c33df8b3cd0a"} Sep 10 19:19:57.309360 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:57.309311 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl" event={"ID":"0ffb7da0-bdf2-4cf1-8fd9-82372e27409f","Type":"ContainerStarted","Data":"06e4b3f02047dc598d16c1eafd3560cb044f5b2c16b22692360a8daf6b710253"} Sep 10 19:19:57.335648 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:19:57.335589 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-wb6hl" podStartSLOduration=1.59944057 podStartE2EDuration="4.335573073s" totalImagesPullingTime="2.736132503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:19:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:19:57.334419879 +0000 UTC m=+246.464921521" watchObservedRunningTime="2026-09-10 19:19:57.335573073 +0000 UTC m=+246.466074707" Sep 10 19:20:35.000866 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.000831 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8"] Sep 10 19:20:35.006285 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.006263 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.008934 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.008908 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 19:20:35.009046 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.008953 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 19:20:35.009089 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.009059 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4qpwt\"" Sep 10 19:20:35.009280 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.009266 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 19:20:35.009330 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.009308 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 19:20:35.020427 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.020407 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8"] Sep 10 19:20:35.168258 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.168225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e66ef0f5-d717-48c0-a395-a5001bf4b377-tmp\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.168433 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.168266 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e66ef0f5-d717-48c0-a395-a5001bf4b377-manifests\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.168433 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.168294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fngmk\" (UniqueName: \"kubernetes.io/projected/e66ef0f5-d717-48c0-a395-a5001bf4b377-kube-api-access-fngmk\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.168433 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.168376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e66ef0f5-d717-48c0-a395-a5001bf4b377-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.168433 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.168418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e66ef0f5-d717-48c0-a395-a5001bf4b377-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.269215 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.269135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e66ef0f5-d717-48c0-a395-a5001bf4b377-tmp\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.269215 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.269179 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e66ef0f5-d717-48c0-a395-a5001bf4b377-manifests\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.269215 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.269208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fngmk\" (UniqueName: \"kubernetes.io/projected/e66ef0f5-d717-48c0-a395-a5001bf4b377-kube-api-access-fngmk\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.269427 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.269240 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e66ef0f5-d717-48c0-a395-a5001bf4b377-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.269427 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.269271 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e66ef0f5-d717-48c0-a395-a5001bf4b377-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.269657 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.269623 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e66ef0f5-d717-48c0-a395-a5001bf4b377-manifests\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.271479 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.271458 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e66ef0f5-d717-48c0-a395-a5001bf4b377-tmp\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.271786 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.271766 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e66ef0f5-d717-48c0-a395-a5001bf4b377-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.271835 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.271817 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e66ef0f5-d717-48c0-a395-a5001bf4b377-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.288135 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.288103 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fngmk\" (UniqueName: \"kubernetes.io/projected/e66ef0f5-d717-48c0-a395-a5001bf4b377-kube-api-access-fngmk\") pod \"opendatahub-operator-controller-manager-5d477566c-rcth8\" (UID: \"e66ef0f5-d717-48c0-a395-a5001bf4b377\") " pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.316364 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.316340 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:35.594051 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:20:35.594020 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66ef0f5_d717_48c0_a395_a5001bf4b377.slice/crio-c6953865251c399e6b009009d79cbe9b76f8cae3a7a22d1ff7ddc9e3a176dbb9 WatchSource:0}: Error finding container c6953865251c399e6b009009d79cbe9b76f8cae3a7a22d1ff7ddc9e3a176dbb9: Status 404 returned error can't find the container with id c6953865251c399e6b009009d79cbe9b76f8cae3a7a22d1ff7ddc9e3a176dbb9 Sep 10 19:20:35.600206 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:35.600179 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8"] Sep 10 19:20:36.446863 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:36.446821 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" event={"ID":"e66ef0f5-d717-48c0-a395-a5001bf4b377","Type":"ContainerStarted","Data":"c6953865251c399e6b009009d79cbe9b76f8cae3a7a22d1ff7ddc9e3a176dbb9"} Sep 10 19:20:39.460549 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:39.460506 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" event={"ID":"e66ef0f5-d717-48c0-a395-a5001bf4b377","Type":"ContainerStarted","Data":"738f0506972e1cfb49d1f50bd967ad2459735496da38bc19e73ba571e3d61c64"} Sep 10 19:20:40.465660 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:40.465622 2578 generic.go:356] "Generic (PLEG): container finished" podID="e66ef0f5-d717-48c0-a395-a5001bf4b377" containerID="738f0506972e1cfb49d1f50bd967ad2459735496da38bc19e73ba571e3d61c64" exitCode=0 Sep 10 19:20:40.466138 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:40.465682 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" event={"ID":"e66ef0f5-d717-48c0-a395-a5001bf4b377","Type":"ContainerDied","Data":"738f0506972e1cfb49d1f50bd967ad2459735496da38bc19e73ba571e3d61c64"} Sep 10 19:20:41.471155 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:41.471122 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" event={"ID":"e66ef0f5-d717-48c0-a395-a5001bf4b377","Type":"ContainerStarted","Data":"b7243c885ae86a3640abf522c9e586a1e80f9d591c91c6c6d74f7d76edf6ad3c"} Sep 10 19:20:41.471520 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:41.471228 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:41.493699 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:41.493648 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" podStartSLOduration=2.312715929 podStartE2EDuration="7.493633982s" totalImagesPullingTime="4.174868232s" totalInitContainerRuntime="1.006049821s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:20:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:41.492468607 +0000 UTC m=+290.622970237" watchObservedRunningTime="2026-09-10 19:20:41.493633982 +0000 UTC m=+290.624135612" Sep 10 19:20:51.205802 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:51.205779 2578 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 19:20:52.476524 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:52.476487 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d477566c-rcth8" Sep 10 19:20:53.995055 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:53.995022 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s"] Sep 10 19:20:53.997972 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:53.997948 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.001000 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.000968 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 19:20:54.001138 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.001003 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 19:20:54.001138 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.001044 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bmgfg\"" Sep 10 19:20:54.001138 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.001045 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 19:20:54.001138 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.001047 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 19:20:54.001138 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.001106 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 19:20:54.011742 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.011707 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s"] Sep 10 19:20:54.113024 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.112990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bacc8f17-9307-4b60-bedc-e236fa0281ac-manager-config\") pod \"lws-controller-manager-b5db5bfb7-f9k4s\" (UID: \"bacc8f17-9307-4b60-bedc-e236fa0281ac\") " pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.113194 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.113041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bacc8f17-9307-4b60-bedc-e236fa0281ac-metrics-cert\") pod \"lws-controller-manager-b5db5bfb7-f9k4s\" (UID: \"bacc8f17-9307-4b60-bedc-e236fa0281ac\") " pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.113194 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.113157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bacc8f17-9307-4b60-bedc-e236fa0281ac-cert\") pod \"lws-controller-manager-b5db5bfb7-f9k4s\" (UID: \"bacc8f17-9307-4b60-bedc-e236fa0281ac\") " pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.113194 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.113185 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dzk5\" (UniqueName: \"kubernetes.io/projected/bacc8f17-9307-4b60-bedc-e236fa0281ac-kube-api-access-5dzk5\") pod \"lws-controller-manager-b5db5bfb7-f9k4s\" (UID: \"bacc8f17-9307-4b60-bedc-e236fa0281ac\") " pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.214328 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.214295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bacc8f17-9307-4b60-bedc-e236fa0281ac-cert\") pod \"lws-controller-manager-b5db5bfb7-f9k4s\" (UID: \"bacc8f17-9307-4b60-bedc-e236fa0281ac\") " pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.214328 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.214328 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dzk5\" (UniqueName: \"kubernetes.io/projected/bacc8f17-9307-4b60-bedc-e236fa0281ac-kube-api-access-5dzk5\") pod \"lws-controller-manager-b5db5bfb7-f9k4s\" (UID: \"bacc8f17-9307-4b60-bedc-e236fa0281ac\") " pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.214559 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.214358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bacc8f17-9307-4b60-bedc-e236fa0281ac-manager-config\") pod \"lws-controller-manager-b5db5bfb7-f9k4s\" (UID: \"bacc8f17-9307-4b60-bedc-e236fa0281ac\") " pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.214559 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.214375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bacc8f17-9307-4b60-bedc-e236fa0281ac-metrics-cert\") pod \"lws-controller-manager-b5db5bfb7-f9k4s\" (UID: \"bacc8f17-9307-4b60-bedc-e236fa0281ac\") " pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.215006 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.214958 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bacc8f17-9307-4b60-bedc-e236fa0281ac-manager-config\") pod \"lws-controller-manager-b5db5bfb7-f9k4s\" (UID: \"bacc8f17-9307-4b60-bedc-e236fa0281ac\") " pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.216931 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.216907 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bacc8f17-9307-4b60-bedc-e236fa0281ac-cert\") pod \"lws-controller-manager-b5db5bfb7-f9k4s\" (UID: \"bacc8f17-9307-4b60-bedc-e236fa0281ac\") " pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.217055 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.217034 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bacc8f17-9307-4b60-bedc-e236fa0281ac-metrics-cert\") pod \"lws-controller-manager-b5db5bfb7-f9k4s\" (UID: \"bacc8f17-9307-4b60-bedc-e236fa0281ac\") " pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.251200 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.251124 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dzk5\" (UniqueName: \"kubernetes.io/projected/bacc8f17-9307-4b60-bedc-e236fa0281ac-kube-api-access-5dzk5\") pod \"lws-controller-manager-b5db5bfb7-f9k4s\" (UID: \"bacc8f17-9307-4b60-bedc-e236fa0281ac\") " pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.306917 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.306887 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:54.588559 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.588534 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s"] Sep 10 19:20:54.590783 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:20:54.590754 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbacc8f17_9307_4b60_bedc_e236fa0281ac.slice/crio-416cc46feebdedf648bafa499b7058ff4ad4ee07726ee2f11a4f4998394d8810 WatchSource:0}: Error finding container 416cc46feebdedf648bafa499b7058ff4ad4ee07726ee2f11a4f4998394d8810: Status 404 returned error can't find the container with id 416cc46feebdedf648bafa499b7058ff4ad4ee07726ee2f11a4f4998394d8810 Sep 10 19:20:54.592909 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:54.592892 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 19:20:55.530090 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:55.530057 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" event={"ID":"bacc8f17-9307-4b60-bedc-e236fa0281ac","Type":"ContainerStarted","Data":"416cc46feebdedf648bafa499b7058ff4ad4ee07726ee2f11a4f4998394d8810"} Sep 10 19:20:57.539649 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:57.539610 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" event={"ID":"bacc8f17-9307-4b60-bedc-e236fa0281ac","Type":"ContainerStarted","Data":"658368e68c8a9c9ebba4e7ddb238ddc371169b9f7bcbf1b82f76b5fce7937e2e"} Sep 10 19:20:57.540019 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:57.539767 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:20:57.557958 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:20:57.557914 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" podStartSLOduration=1.75596847 podStartE2EDuration="4.55789725s" totalImagesPullingTime="2.80192878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:20:57.556450939 +0000 UTC m=+306.686952569" watchObservedRunningTime="2026-09-10 19:20:57.55789725 +0000 UTC m=+306.688398882" Sep 10 19:21:08.545552 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:08.545523 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-b5db5bfb7-f9k4s" Sep 10 19:21:12.432498 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.432455 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x"] Sep 10 19:21:12.461963 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.461931 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x"] Sep 10 19:21:12.462109 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.462030 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.465216 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.465191 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 19:21:12.465437 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.465419 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 19:21:12.465885 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.465868 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 19:21:12.466060 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.466047 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 19:21:12.466529 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.466507 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2rv4h\"" Sep 10 19:21:12.554394 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.554355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eb01b46a-180c-4163-9c01-c219b7b2ada9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.554569 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.554403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eb01b46a-180c-4163-9c01-c219b7b2ada9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.554569 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.554433 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb01b46a-180c-4163-9c01-c219b7b2ada9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.554569 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.554498 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q278p\" (UniqueName: \"kubernetes.io/projected/eb01b46a-180c-4163-9c01-c219b7b2ada9-kube-api-access-q278p\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.554569 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.554542 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eb01b46a-180c-4163-9c01-c219b7b2ada9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.554569 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.554566 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eb01b46a-180c-4163-9c01-c219b7b2ada9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.554750 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.554597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb01b46a-180c-4163-9c01-c219b7b2ada9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.655250 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.655211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eb01b46a-180c-4163-9c01-c219b7b2ada9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.655421 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.655272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eb01b46a-180c-4163-9c01-c219b7b2ada9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.655421 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.655313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb01b46a-180c-4163-9c01-c219b7b2ada9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.655421 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.655359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q278p\" (UniqueName: \"kubernetes.io/projected/eb01b46a-180c-4163-9c01-c219b7b2ada9-kube-api-access-q278p\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.655421 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.655405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eb01b46a-180c-4163-9c01-c219b7b2ada9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.655605 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.655431 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eb01b46a-180c-4163-9c01-c219b7b2ada9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.655605 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.655460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb01b46a-180c-4163-9c01-c219b7b2ada9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.656020 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.655987 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/eb01b46a-180c-4163-9c01-c219b7b2ada9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.657980 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.657951 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb01b46a-180c-4163-9c01-c219b7b2ada9-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.658086 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.658028 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/eb01b46a-180c-4163-9c01-c219b7b2ada9-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.658086 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.658034 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/eb01b46a-180c-4163-9c01-c219b7b2ada9-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.658251 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.658218 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/eb01b46a-180c-4163-9c01-c219b7b2ada9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.664327 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.664307 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb01b46a-180c-4163-9c01-c219b7b2ada9-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.664477 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.664456 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q278p\" (UniqueName: \"kubernetes.io/projected/eb01b46a-180c-4163-9c01-c219b7b2ada9-kube-api-access-q278p\") pod \"istiod-openshift-gateway-5677ccb49b-p596x\" (UID: \"eb01b46a-180c-4163-9c01-c219b7b2ada9\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:12.771698 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:12.771612 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:13.054156 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:13.054134 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x"] Sep 10 19:21:13.056160 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:21:13.056126 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb01b46a_180c_4163_9c01_c219b7b2ada9.slice/crio-bf3534ff421f0ff8966d917d8689db28b81c7d0f16f00d75b63fdb132ed9b6d3 WatchSource:0}: Error finding container bf3534ff421f0ff8966d917d8689db28b81c7d0f16f00d75b63fdb132ed9b6d3: Status 404 returned error can't find the container with id bf3534ff421f0ff8966d917d8689db28b81c7d0f16f00d75b63fdb132ed9b6d3 Sep 10 19:21:13.604533 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:13.604451 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" event={"ID":"eb01b46a-180c-4163-9c01-c219b7b2ada9","Type":"ContainerStarted","Data":"bf3534ff421f0ff8966d917d8689db28b81c7d0f16f00d75b63fdb132ed9b6d3"} Sep 10 19:21:15.387822 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:15.387757 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 19:21:15.388102 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:15.387862 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"} Sep 10 19:21:15.619170 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:15.619129 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" event={"ID":"eb01b46a-180c-4163-9c01-c219b7b2ada9","Type":"ContainerStarted","Data":"04e99c5c9c7b45e7cec2882085e2ef1f8a6ee109a49d456e86c632dfe5f2e40d"} Sep 10 19:21:15.619349 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:15.619255 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:21:15.679506 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:15.679400 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" podStartSLOduration=1.350328018 podStartE2EDuration="3.679382627s" totalImagesPullingTime="2.329054609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:21:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:21:15.676351286 +0000 UTC m=+324.806852915" watchObservedRunningTime="2026-09-10 19:21:15.679382627 +0000 UTC m=+324.809884258" Sep 10 19:21:16.625541 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:21:16.625504 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-p596x" Sep 10 19:22:47.464497 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.464444 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj"] Sep 10 19:22:47.469036 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.469008 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" Sep 10 19:22:47.471317 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.471284 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 19:22:47.471567 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.471539 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 19:22:47.471650 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.471618 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2r8zh\"" Sep 10 19:22:47.483314 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.483287 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj"] Sep 10 19:22:47.544614 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.544580 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj"] Sep 10 19:22:47.544863 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:22:47.544842 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-zn5q9], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" podUID="341e1644-2e08-4fda-ae8f-233aeb0dbb92" Sep 10 19:22:47.556856 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.556827 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj"] Sep 10 19:22:47.571222 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.571194 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn"] Sep 10 19:22:47.574424 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.574406 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" Sep 10 19:22:47.591411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.591387 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn"] Sep 10 19:22:47.600311 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.600278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/341e1644-2e08-4fda-ae8f-233aeb0dbb92-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w8bjj\" (UID: \"341e1644-2e08-4fda-ae8f-233aeb0dbb92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" Sep 10 19:22:47.600452 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.600344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn5q9\" (UniqueName: \"kubernetes.io/projected/341e1644-2e08-4fda-ae8f-233aeb0dbb92-kube-api-access-zn5q9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w8bjj\" (UID: \"341e1644-2e08-4fda-ae8f-233aeb0dbb92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" Sep 10 19:22:47.700892 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.700853 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9wqp\" (UniqueName: \"kubernetes.io/projected/24f1bba8-35ab-4369-a146-582ceef19c0f-kube-api-access-k9wqp\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kn8qn\" (UID: \"24f1bba8-35ab-4369-a146-582ceef19c0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" Sep 10 19:22:47.701067 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.700904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn5q9\" (UniqueName: \"kubernetes.io/projected/341e1644-2e08-4fda-ae8f-233aeb0dbb92-kube-api-access-zn5q9\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w8bjj\" (UID: \"341e1644-2e08-4fda-ae8f-233aeb0dbb92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" Sep 10 19:22:47.701067 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.701003 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24f1bba8-35ab-4369-a146-582ceef19c0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kn8qn\" (UID: \"24f1bba8-35ab-4369-a146-582ceef19c0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" Sep 10 19:22:47.701067 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.701040 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/341e1644-2e08-4fda-ae8f-233aeb0dbb92-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w8bjj\" (UID: \"341e1644-2e08-4fda-ae8f-233aeb0dbb92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" Sep 10 19:22:47.701386 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.701369 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/341e1644-2e08-4fda-ae8f-233aeb0dbb92-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-w8bjj\" (UID: \"341e1644-2e08-4fda-ae8f-233aeb0dbb92\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" Sep 10 19:22:47.704921 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:22:47.704905 2578 projected.go:196] Error preparing data for projected volume kube-api-access-zn5q9 for pod kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj: failed to fetch token: pod "kuadrant-operator-controller-manager-5769f99d9f-w8bjj" not found Sep 10 19:22:47.705007 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:22:47.704973 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/341e1644-2e08-4fda-ae8f-233aeb0dbb92-kube-api-access-zn5q9 podName:341e1644-2e08-4fda-ae8f-233aeb0dbb92 nodeName:}" failed. No retries permitted until 2026-09-10 19:22:48.204954833 +0000 UTC m=+417.335456441 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zn5q9" (UniqueName: "kubernetes.io/projected/341e1644-2e08-4fda-ae8f-233aeb0dbb92-kube-api-access-zn5q9") pod "kuadrant-operator-controller-manager-5769f99d9f-w8bjj" (UID: "341e1644-2e08-4fda-ae8f-233aeb0dbb92") : failed to fetch token: pod "kuadrant-operator-controller-manager-5769f99d9f-w8bjj" not found Sep 10 19:22:47.801549 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.801473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9wqp\" (UniqueName: \"kubernetes.io/projected/24f1bba8-35ab-4369-a146-582ceef19c0f-kube-api-access-k9wqp\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kn8qn\" (UID: \"24f1bba8-35ab-4369-a146-582ceef19c0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" Sep 10 19:22:47.801690 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.801576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24f1bba8-35ab-4369-a146-582ceef19c0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kn8qn\" (UID: \"24f1bba8-35ab-4369-a146-582ceef19c0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" Sep 10 19:22:47.801956 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.801935 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24f1bba8-35ab-4369-a146-582ceef19c0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kn8qn\" (UID: \"24f1bba8-35ab-4369-a146-582ceef19c0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" Sep 10 19:22:47.810516 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.810494 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9wqp\" (UniqueName: \"kubernetes.io/projected/24f1bba8-35ab-4369-a146-582ceef19c0f-kube-api-access-k9wqp\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kn8qn\" (UID: \"24f1bba8-35ab-4369-a146-582ceef19c0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" Sep 10 19:22:47.884517 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.884493 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" Sep 10 19:22:47.982360 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.982331 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" Sep 10 19:22:47.984456 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:22:47.984426 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-w8bjj\" is forbidden: User \"system:node:ip-10-0-138-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-71.ec2.internal' and this object" podUID="341e1644-2e08-4fda-ae8f-233aeb0dbb92" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" Sep 10 19:22:47.989294 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:47.989273 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" Sep 10 19:22:47.991166 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:22:47.991142 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-w8bjj\" is forbidden: User \"system:node:ip-10-0-138-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-71.ec2.internal' and this object" podUID="341e1644-2e08-4fda-ae8f-233aeb0dbb92" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" Sep 10 19:22:48.104649 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:48.104587 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/341e1644-2e08-4fda-ae8f-233aeb0dbb92-extensions-socket-volume\") pod \"341e1644-2e08-4fda-ae8f-233aeb0dbb92\" (UID: \"341e1644-2e08-4fda-ae8f-233aeb0dbb92\") " Sep 10 19:22:48.104788 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:48.104707 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn5q9\" (UniqueName: \"kubernetes.io/projected/341e1644-2e08-4fda-ae8f-233aeb0dbb92-kube-api-access-zn5q9\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:22:48.104867 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:48.104846 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/341e1644-2e08-4fda-ae8f-233aeb0dbb92-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "341e1644-2e08-4fda-ae8f-233aeb0dbb92" (UID: "341e1644-2e08-4fda-ae8f-233aeb0dbb92"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:22:48.162244 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:48.162215 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn"] Sep 10 19:22:48.166108 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:22:48.166084 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f1bba8_35ab_4369_a146_582ceef19c0f.slice/crio-bcbd6177c4a07ff958671cbae16443d50bd0a235d067c0a738aa44bb71c90448 WatchSource:0}: Error finding container bcbd6177c4a07ff958671cbae16443d50bd0a235d067c0a738aa44bb71c90448: Status 404 returned error can't find the container with id bcbd6177c4a07ff958671cbae16443d50bd0a235d067c0a738aa44bb71c90448 Sep 10 19:22:48.205647 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:48.205622 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/341e1644-2e08-4fda-ae8f-233aeb0dbb92-extensions-socket-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:22:48.989676 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:48.989641 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" Sep 10 19:22:48.990169 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:48.989706 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" event={"ID":"24f1bba8-35ab-4369-a146-582ceef19c0f","Type":"ContainerStarted","Data":"bcbd6177c4a07ff958671cbae16443d50bd0a235d067c0a738aa44bb71c90448"} Sep 10 19:22:48.992445 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:22:48.992412 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-w8bjj\" is forbidden: User \"system:node:ip-10-0-138-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-71.ec2.internal' and this object" podUID="341e1644-2e08-4fda-ae8f-233aeb0dbb92" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" Sep 10 19:22:49.000664 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:22:49.000636 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-w8bjj\" is forbidden: User \"system:node:ip-10-0-138-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-71.ec2.internal' and this object" podUID="341e1644-2e08-4fda-ae8f-233aeb0dbb92" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-w8bjj" Sep 10 19:22:49.392825 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:49.392792 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="341e1644-2e08-4fda-ae8f-233aeb0dbb92" path="/var/lib/kubelet/pods/341e1644-2e08-4fda-ae8f-233aeb0dbb92/volumes" Sep 10 19:22:53.009898 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:53.009867 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" event={"ID":"24f1bba8-35ab-4369-a146-582ceef19c0f","Type":"ContainerStarted","Data":"0ab6a87b637349d9a21edcba038c8a607e8abf3df33ae139ea3f0fd97d3892c0"} Sep 10 19:22:53.010318 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:53.010001 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" Sep 10 19:22:53.031577 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:22:53.031521 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" podStartSLOduration=2.187472894 podStartE2EDuration="6.031502426s" totalImagesPullingTime="3.844029532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:22:53.02969236 +0000 UTC m=+422.160193988" watchObservedRunningTime="2026-09-10 19:22:53.031502426 +0000 UTC m=+422.162004057" Sep 10 19:23:04.016499 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:23:04.016462 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" Sep 10 19:24:34.926319 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:34.926280 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-wtc2c"] Sep 10 19:24:34.929948 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:34.929925 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-wtc2c" Sep 10 19:24:34.934394 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:34.934359 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-v6525\"" Sep 10 19:24:34.934708 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:34.934688 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 10 19:24:34.942215 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:34.942191 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-wtc2c"] Sep 10 19:24:35.083704 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:35.083663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5bwl\" (UniqueName: \"kubernetes.io/projected/73d78e35-5e50-40d2-b50c-ed3ff13f031e-kube-api-access-s5bwl\") pod \"payload-processing-5b54b7fbb7-wtc2c\" (UID: \"73d78e35-5e50-40d2-b50c-ed3ff13f031e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wtc2c" Sep 10 19:24:35.083913 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:35.083758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d78e35-5e50-40d2-b50c-ed3ff13f031e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-wtc2c\" (UID: \"73d78e35-5e50-40d2-b50c-ed3ff13f031e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wtc2c" Sep 10 19:24:35.184552 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:35.184463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d78e35-5e50-40d2-b50c-ed3ff13f031e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-wtc2c\" (UID: \"73d78e35-5e50-40d2-b50c-ed3ff13f031e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wtc2c" Sep 10 19:24:35.184552 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:35.184549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5bwl\" (UniqueName: \"kubernetes.io/projected/73d78e35-5e50-40d2-b50c-ed3ff13f031e-kube-api-access-s5bwl\") pod \"payload-processing-5b54b7fbb7-wtc2c\" (UID: \"73d78e35-5e50-40d2-b50c-ed3ff13f031e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wtc2c" Sep 10 19:24:35.185218 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:35.185197 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73d78e35-5e50-40d2-b50c-ed3ff13f031e-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-wtc2c\" (UID: \"73d78e35-5e50-40d2-b50c-ed3ff13f031e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wtc2c" Sep 10 19:24:35.193391 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:35.193360 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5bwl\" (UniqueName: \"kubernetes.io/projected/73d78e35-5e50-40d2-b50c-ed3ff13f031e-kube-api-access-s5bwl\") pod \"payload-processing-5b54b7fbb7-wtc2c\" (UID: \"73d78e35-5e50-40d2-b50c-ed3ff13f031e\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-wtc2c" Sep 10 19:24:35.244753 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:35.244688 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-wtc2c" Sep 10 19:24:35.516472 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:35.516445 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-wtc2c"] Sep 10 19:24:35.519263 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:24:35.519234 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73d78e35_5e50_40d2_b50c_ed3ff13f031e.slice/crio-402f180ccf691dd0a174502c9a8475a758dc8d86d528789c87a26b73223eadf6 WatchSource:0}: Error finding container 402f180ccf691dd0a174502c9a8475a758dc8d86d528789c87a26b73223eadf6: Status 404 returned error can't find the container with id 402f180ccf691dd0a174502c9a8475a758dc8d86d528789c87a26b73223eadf6 Sep 10 19:24:36.432288 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:36.432240 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-wtc2c" event={"ID":"73d78e35-5e50-40d2-b50c-ed3ff13f031e","Type":"ContainerStarted","Data":"402f180ccf691dd0a174502c9a8475a758dc8d86d528789c87a26b73223eadf6"} Sep 10 19:24:37.438503 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:37.438455 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-wtc2c" event={"ID":"73d78e35-5e50-40d2-b50c-ed3ff13f031e","Type":"ContainerStarted","Data":"aae2397d495c22ee9cb7eaabf1a74b0d43dccc561329ac8c8a55015de009fc27"} Sep 10 19:24:37.438998 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:37.438567 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-wtc2c" Sep 10 19:24:37.457006 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:37.456956 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-wtc2c" podStartSLOduration=1.755605482 podStartE2EDuration="3.456938329s" totalImagesPullingTime="1.701332847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:24:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:24:37.455418629 +0000 UTC m=+526.585920260" watchObservedRunningTime="2026-09-10 19:24:37.456938329 +0000 UTC m=+526.587439962" Sep 10 19:24:48.444998 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:24:48.444913 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-wtc2c" Sep 10 19:25:40.372347 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.372315 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2"] Sep 10 19:25:40.375947 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.375929 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.379245 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.379224 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 10 19:25:40.379245 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.379237 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 19:25:40.379432 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.379286 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 19:25:40.379432 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.379237 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6nnj8\"" Sep 10 19:25:40.390875 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.390855 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2"] Sep 10 19:25:40.418121 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.418095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887a4e2a-40b5-448a-82f1-b28c7677a674-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.418227 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.418154 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887a4e2a-40b5-448a-82f1-b28c7677a674-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.418227 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.418196 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887a4e2a-40b5-448a-82f1-b28c7677a674-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.418309 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.418244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887a4e2a-40b5-448a-82f1-b28c7677a674-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.418309 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.418270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqjwk\" (UniqueName: \"kubernetes.io/projected/887a4e2a-40b5-448a-82f1-b28c7677a674-kube-api-access-wqjwk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.418435 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.418396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887a4e2a-40b5-448a-82f1-b28c7677a674-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.519089 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.519057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887a4e2a-40b5-448a-82f1-b28c7677a674-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.519089 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.519090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887a4e2a-40b5-448a-82f1-b28c7677a674-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.519298 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.519121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887a4e2a-40b5-448a-82f1-b28c7677a674-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.519298 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.519138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887a4e2a-40b5-448a-82f1-b28c7677a674-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.519298 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.519181 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887a4e2a-40b5-448a-82f1-b28c7677a674-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.519298 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.519208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqjwk\" (UniqueName: \"kubernetes.io/projected/887a4e2a-40b5-448a-82f1-b28c7677a674-kube-api-access-wqjwk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.519515 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.519491 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/887a4e2a-40b5-448a-82f1-b28c7677a674-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.519570 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.519554 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/887a4e2a-40b5-448a-82f1-b28c7677a674-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.519570 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.519560 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/887a4e2a-40b5-448a-82f1-b28c7677a674-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.521636 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.521612 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/887a4e2a-40b5-448a-82f1-b28c7677a674-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.521779 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.521763 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/887a4e2a-40b5-448a-82f1-b28c7677a674-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.527122 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.527100 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqjwk\" (UniqueName: \"kubernetes.io/projected/887a4e2a-40b5-448a-82f1-b28c7677a674-kube-api-access-wqjwk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h9dp2\" (UID: \"887a4e2a-40b5-448a-82f1-b28c7677a674\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.686449 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.686373 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:40.958876 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:40.958849 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2"] Sep 10 19:25:40.961200 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:25:40.961167 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887a4e2a_40b5_448a_82f1_b28c7677a674.slice/crio-b50f95107deca33e720837a9307aafd6d3a80122520ba8b450405f1f53e2ee08 WatchSource:0}: Error finding container b50f95107deca33e720837a9307aafd6d3a80122520ba8b450405f1f53e2ee08: Status 404 returned error can't find the container with id b50f95107deca33e720837a9307aafd6d3a80122520ba8b450405f1f53e2ee08 Sep 10 19:25:41.695656 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:41.695619 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" event={"ID":"887a4e2a-40b5-448a-82f1-b28c7677a674","Type":"ContainerStarted","Data":"b50f95107deca33e720837a9307aafd6d3a80122520ba8b450405f1f53e2ee08"} Sep 10 19:25:42.701853 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:42.701818 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" event={"ID":"887a4e2a-40b5-448a-82f1-b28c7677a674","Type":"ContainerStarted","Data":"9841fc995b406bfa94d752dfc9f34767cd9570db944b4c5f2a3ca8447aca4bb9"} Sep 10 19:25:42.720566 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:42.720515 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" podStartSLOduration=1.754182766 podStartE2EDuration="2.720490439s" totalImagesPullingTime="966.307673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:25:42.718975943 +0000 UTC m=+591.849477573" watchObservedRunningTime="2026-09-10 19:25:42.720490439 +0000 UTC m=+591.850992069" Sep 10 19:25:50.686657 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:50.686617 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:50.687057 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:50.686702 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:50.698845 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:50.698823 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:50.746036 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:50.746013 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h9dp2" Sep 10 19:25:54.173641 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.173599 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql"] Sep 10 19:25:54.176682 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.176667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.179280 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.179259 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 10 19:25:54.190094 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.190070 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql"] Sep 10 19:25:54.222530 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.222505 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.222648 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.222540 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.222648 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.222576 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.222801 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.222658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.222801 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.222705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5ptt\" (UniqueName: \"kubernetes.io/projected/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-kube-api-access-x5ptt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.222801 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.222765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.323490 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.323461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.323641 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.323497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5ptt\" (UniqueName: \"kubernetes.io/projected/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-kube-api-access-x5ptt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.323641 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.323518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.323641 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.323556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.323641 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.323594 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.323900 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.323781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.323973 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.323951 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.324020 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.324001 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.324140 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.324113 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.325956 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.325936 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.326313 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.326293 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.331335 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.331317 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5ptt\" (UniqueName: \"kubernetes.io/projected/4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca-kube-api-access-x5ptt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql\" (UID: \"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.486680 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.486604 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:25:54.757359 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.757337 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql"] Sep 10 19:25:54.758930 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:25:54.758903 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbe0ddb_33c9_4b7e_a1c6_4381dc29c8ca.slice/crio-8b5d4f2ce666e8594ea15c2997ecd19000ff51097cec60196291d723df2616d5 WatchSource:0}: Error finding container 8b5d4f2ce666e8594ea15c2997ecd19000ff51097cec60196291d723df2616d5: Status 404 returned error can't find the container with id 8b5d4f2ce666e8594ea15c2997ecd19000ff51097cec60196291d723df2616d5 Sep 10 19:25:54.760313 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:54.760295 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 19:25:55.757084 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:55.757050 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" event={"ID":"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca","Type":"ContainerStarted","Data":"9e645d0920457d91e707d7b93e9f26458d480c248362cf6b95c727b6e1312faa"} Sep 10 19:25:55.757549 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:55.757106 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" event={"ID":"4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca","Type":"ContainerStarted","Data":"8b5d4f2ce666e8594ea15c2997ecd19000ff51097cec60196291d723df2616d5"} Sep 10 19:25:55.781868 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:25:55.781818 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" podStartSLOduration=1.588590234 podStartE2EDuration="1.781803681s" totalImagesPullingTime="193.213447ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:25:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:25:55.78026165 +0000 UTC m=+604.910763281" watchObservedRunningTime="2026-09-10 19:25:55.781803681 +0000 UTC m=+604.912305311" Sep 10 19:26:04.487314 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:04.487268 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:26:04.487805 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:04.487343 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:26:04.499594 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:04.499574 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:26:04.806140 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:04.806060 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql" Sep 10 19:26:10.991700 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:10.991668 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g"] Sep 10 19:26:10.996199 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:10.996176 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:10.998604 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:10.998584 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 19:26:11.006292 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.006274 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g"] Sep 10 19:26:11.056481 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.056457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.056697 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.056670 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6d6c\" (UniqueName: \"kubernetes.io/projected/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-kube-api-access-j6d6c\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.056841 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.056762 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.056892 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.056845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.056892 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.056881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.056980 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.056901 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.157662 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.157629 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.157662 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.157662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6d6c\" (UniqueName: \"kubernetes.io/projected/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-kube-api-access-j6d6c\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.157849 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.157683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.157894 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.157871 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.157943 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.157927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.157982 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.157962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.158043 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.158026 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.158089 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.158032 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.158302 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.158283 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.160125 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.160102 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.160484 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.160465 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.166085 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.166062 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6d6c\" (UniqueName: \"kubernetes.io/projected/df7cf0ab-0e3a-41c8-8b30-733b3643c7a4-kube-api-access-j6d6c\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w944g\" (UID: \"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.307406 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.307332 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:11.579517 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.579450 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g"] Sep 10 19:26:11.583153 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:26:11.583130 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7cf0ab_0e3a_41c8_8b30_733b3643c7a4.slice/crio-8b878a60a5f11fc9194ab136e17a82f115b2f61533f049c9bc735fcb28a6cac7 WatchSource:0}: Error finding container 8b878a60a5f11fc9194ab136e17a82f115b2f61533f049c9bc735fcb28a6cac7: Status 404 returned error can't find the container with id 8b878a60a5f11fc9194ab136e17a82f115b2f61533f049c9bc735fcb28a6cac7 Sep 10 19:26:11.830283 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:11.830230 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" event={"ID":"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4","Type":"ContainerStarted","Data":"8b878a60a5f11fc9194ab136e17a82f115b2f61533f049c9bc735fcb28a6cac7"} Sep 10 19:26:12.836336 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:12.836301 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" event={"ID":"df7cf0ab-0e3a-41c8-8b30-733b3643c7a4","Type":"ContainerStarted","Data":"44b6f95cfe3686a79e9138a21e23946e6030ba9c7282fd9db2e012b9ff082836"} Sep 10 19:26:12.855526 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:12.855480 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" podStartSLOduration=2.665089246 podStartE2EDuration="2.855465288s" totalImagesPullingTime="190.376042ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:26:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:26:12.854073785 +0000 UTC m=+621.984575416" watchObservedRunningTime="2026-09-10 19:26:12.855465288 +0000 UTC m=+621.985966919" Sep 10 19:26:21.308398 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:21.308358 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:21.308903 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:21.308432 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:21.320752 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:21.320715 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:26:21.888424 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:26:21.888389 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w944g" Sep 10 19:27:48.361991 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:48.361908 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8"] Sep 10 19:27:48.371532 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:48.371499 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" Sep 10 19:27:48.375861 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:48.375841 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8300b9\"" Sep 10 19:27:48.377597 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:48.377569 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8300b9-dockercfg-2bhzh\"" Sep 10 19:27:48.379230 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:48.379210 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8"] Sep 10 19:27:48.438466 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:48.438437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dkxb\" (UniqueName: \"kubernetes.io/projected/f7bfd7b8-44cb-47f2-872d-0ebadfe0af30-kube-api-access-9dkxb\") pod \"payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8\" (UID: \"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" Sep 10 19:27:48.438635 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:48.438483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bfd7b8-44cb-47f2-872d-0ebadfe0af30-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8\" (UID: \"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" Sep 10 19:27:48.539082 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:48.539052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dkxb\" (UniqueName: \"kubernetes.io/projected/f7bfd7b8-44cb-47f2-872d-0ebadfe0af30-kube-api-access-9dkxb\") pod \"payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8\" (UID: \"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" Sep 10 19:27:48.539243 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:48.539102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bfd7b8-44cb-47f2-872d-0ebadfe0af30-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8\" (UID: \"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" Sep 10 19:27:48.539997 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:48.539973 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bfd7b8-44cb-47f2-872d-0ebadfe0af30-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8\" (UID: \"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" Sep 10 19:27:48.547338 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:48.547311 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dkxb\" (UniqueName: \"kubernetes.io/projected/f7bfd7b8-44cb-47f2-872d-0ebadfe0af30-kube-api-access-9dkxb\") pod \"payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8\" (UID: \"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" Sep 10 19:27:48.683183 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:48.683148 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" Sep 10 19:27:48.951639 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:48.951612 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8"] Sep 10 19:27:48.954260 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:27:48.954225 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bfd7b8_44cb_47f2_872d_0ebadfe0af30.slice/crio-bd5ff1f8e7279cfcea8da0c0ed799b1182c3fa57402744de3ed5bff8925623fc WatchSource:0}: Error finding container bd5ff1f8e7279cfcea8da0c0ed799b1182c3fa57402744de3ed5bff8925623fc: Status 404 returned error can't find the container with id bd5ff1f8e7279cfcea8da0c0ed799b1182c3fa57402744de3ed5bff8925623fc Sep 10 19:27:49.244705 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:49.244615 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" event={"ID":"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30","Type":"ContainerStarted","Data":"608803ed4771799749562b199e25efcba80c85197b7578dcf9b5fe812d47868f"} Sep 10 19:27:49.244705 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:49.244674 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" event={"ID":"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30","Type":"ContainerStarted","Data":"bd5ff1f8e7279cfcea8da0c0ed799b1182c3fa57402744de3ed5bff8925623fc"} Sep 10 19:27:49.244705 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:49.244705 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" Sep 10 19:27:49.262702 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:49.262659 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" podStartSLOduration=1.262645423 podStartE2EDuration="1.262645423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:49.261115738 +0000 UTC m=+718.391617370" watchObservedRunningTime="2026-09-10 19:27:49.262645423 +0000 UTC m=+718.393147053" Sep 10 19:27:55.439313 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.439285 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr"] Sep 10 19:27:55.442865 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.442848 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" Sep 10 19:27:55.445128 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.445101 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-b3108b\"" Sep 10 19:27:55.448893 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.448870 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg"] Sep 10 19:27:55.454334 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.454316 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr"] Sep 10 19:27:55.454434 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.454390 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" Sep 10 19:27:55.464460 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.464433 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg"] Sep 10 19:27:55.499386 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.499358 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mkdm\" (UniqueName: \"kubernetes.io/projected/4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10-kube-api-access-6mkdm\") pod \"payload-processing-e2e-autores-b3108b-9449f7f77-z27gg\" (UID: \"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10\") " pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" Sep 10 19:27:55.499524 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.499390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txm56\" (UniqueName: \"kubernetes.io/projected/5c249300-612f-4716-ba16-9d3a7c48c08a-kube-api-access-txm56\") pod \"payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr\" (UID: \"5c249300-612f-4716-ba16-9d3a7c48c08a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" Sep 10 19:27:55.499524 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.499422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c249300-612f-4716-ba16-9d3a7c48c08a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr\" (UID: \"5c249300-612f-4716-ba16-9d3a7c48c08a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" Sep 10 19:27:55.499638 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.499576 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10-plugins-config-volume\") pod \"payload-processing-e2e-autores-b3108b-9449f7f77-z27gg\" (UID: \"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10\") " pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" Sep 10 19:27:55.600366 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.600338 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10-plugins-config-volume\") pod \"payload-processing-e2e-autores-b3108b-9449f7f77-z27gg\" (UID: \"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10\") " pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" Sep 10 19:27:55.600540 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.600383 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mkdm\" (UniqueName: \"kubernetes.io/projected/4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10-kube-api-access-6mkdm\") pod \"payload-processing-e2e-autores-b3108b-9449f7f77-z27gg\" (UID: \"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10\") " pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" Sep 10 19:27:55.600540 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.600404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txm56\" (UniqueName: \"kubernetes.io/projected/5c249300-612f-4716-ba16-9d3a7c48c08a-kube-api-access-txm56\") pod \"payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr\" (UID: \"5c249300-612f-4716-ba16-9d3a7c48c08a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" Sep 10 19:27:55.600540 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.600434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c249300-612f-4716-ba16-9d3a7c48c08a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr\" (UID: \"5c249300-612f-4716-ba16-9d3a7c48c08a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" Sep 10 19:27:55.601312 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.601276 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10-plugins-config-volume\") pod \"payload-processing-e2e-autores-b3108b-9449f7f77-z27gg\" (UID: \"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10\") " pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" Sep 10 19:27:55.601512 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.601485 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c249300-612f-4716-ba16-9d3a7c48c08a-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr\" (UID: \"5c249300-612f-4716-ba16-9d3a7c48c08a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" Sep 10 19:27:55.608807 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.608777 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txm56\" (UniqueName: \"kubernetes.io/projected/5c249300-612f-4716-ba16-9d3a7c48c08a-kube-api-access-txm56\") pod \"payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr\" (UID: \"5c249300-612f-4716-ba16-9d3a7c48c08a\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" Sep 10 19:27:55.609036 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.609017 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mkdm\" (UniqueName: \"kubernetes.io/projected/4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10-kube-api-access-6mkdm\") pod \"payload-processing-e2e-autores-b3108b-9449f7f77-z27gg\" (UID: \"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10\") " pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" Sep 10 19:27:55.754422 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.754336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" Sep 10 19:27:55.765632 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:55.765610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" Sep 10 19:27:56.047416 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:56.047393 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr"] Sep 10 19:27:56.049349 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:27:56.049323 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c249300_612f_4716_ba16_9d3a7c48c08a.slice/crio-4f4429bdc338ae2185283fc8f1bcd996654bec004884f4acf9c79895a7f98fb9 WatchSource:0}: Error finding container 4f4429bdc338ae2185283fc8f1bcd996654bec004884f4acf9c79895a7f98fb9: Status 404 returned error can't find the container with id 4f4429bdc338ae2185283fc8f1bcd996654bec004884f4acf9c79895a7f98fb9 Sep 10 19:27:56.054437 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:56.054415 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg"] Sep 10 19:27:56.056019 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:27:56.055997 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b9b72f5_dc34_4fad_81fb_8a0c2a5faf10.slice/crio-9908112db1863bf7e86f98717fad75d9ddfca5c223acd1b4c21da4ad91906178 WatchSource:0}: Error finding container 9908112db1863bf7e86f98717fad75d9ddfca5c223acd1b4c21da4ad91906178: Status 404 returned error can't find the container with id 9908112db1863bf7e86f98717fad75d9ddfca5c223acd1b4c21da4ad91906178 Sep 10 19:27:56.278832 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:56.278794 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" event={"ID":"5c249300-612f-4716-ba16-9d3a7c48c08a","Type":"ContainerStarted","Data":"ffaf65659fdb78966bcea6d00b3d2ff514a13c6307763019dfb167f36fa55143"} Sep 10 19:27:56.279022 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:56.278862 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" event={"ID":"5c249300-612f-4716-ba16-9d3a7c48c08a","Type":"ContainerStarted","Data":"4f4429bdc338ae2185283fc8f1bcd996654bec004884f4acf9c79895a7f98fb9"} Sep 10 19:27:56.279022 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:56.278893 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" Sep 10 19:27:56.280519 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:56.280486 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" event={"ID":"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10","Type":"ContainerStarted","Data":"8f9c486537dc3cac9f94d8636836d6369b2f7a80c6b08880e9a8ccae2227bdfb"} Sep 10 19:27:56.280603 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:56.280534 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" event={"ID":"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10","Type":"ContainerStarted","Data":"9908112db1863bf7e86f98717fad75d9ddfca5c223acd1b4c21da4ad91906178"} Sep 10 19:27:56.280603 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:56.280556 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" Sep 10 19:27:56.296981 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:56.296882 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" podStartSLOduration=1.296866935 podStartE2EDuration="1.296866935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:56.294945792 +0000 UTC m=+725.425447427" watchObservedRunningTime="2026-09-10 19:27:56.296866935 +0000 UTC m=+725.427368566" Sep 10 19:27:56.312362 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:27:56.312317 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" podStartSLOduration=1.312304238 podStartE2EDuration="1.312304238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:27:56.310659311 +0000 UTC m=+725.441160944" watchObservedRunningTime="2026-09-10 19:27:56.312304238 +0000 UTC m=+725.442805882" Sep 10 19:28:00.251672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:00.251641 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" Sep 10 19:28:00.542032 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:00.541962 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc"] Sep 10 19:28:00.545662 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:00.545641 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" Sep 10 19:28:00.551060 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:28:00.551026 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc\" is forbidden: User \"system:node:ip-10-0-138-71.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-71.ec2.internal' and this object" podUID="db1cb268-bac3-4654-9fc2-b3cef57d36de" pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" Sep 10 19:28:00.551163 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:28:00.551048 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-ait-85e90bf8-dockercfg-nnl6w\" is forbidden: User \"system:node:ip-10-0-138-71.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-138-71.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-85e90bf8-dockercfg-nnl6w\"" type="*v1.Secret" Sep 10 19:28:00.561132 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:00.561113 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-85e90bf8\"" Sep 10 19:28:00.572599 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:00.572577 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc"] Sep 10 19:28:00.648207 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:00.648175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1cb268-bac3-4654-9fc2-b3cef57d36de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc\" (UID: \"db1cb268-bac3-4654-9fc2-b3cef57d36de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" Sep 10 19:28:00.648317 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:00.648221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4dmk\" (UniqueName: \"kubernetes.io/projected/db1cb268-bac3-4654-9fc2-b3cef57d36de-kube-api-access-f4dmk\") pod \"payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc\" (UID: \"db1cb268-bac3-4654-9fc2-b3cef57d36de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" Sep 10 19:28:00.749427 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:00.749394 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1cb268-bac3-4654-9fc2-b3cef57d36de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc\" (UID: \"db1cb268-bac3-4654-9fc2-b3cef57d36de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" Sep 10 19:28:00.749566 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:00.749443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4dmk\" (UniqueName: \"kubernetes.io/projected/db1cb268-bac3-4654-9fc2-b3cef57d36de-kube-api-access-f4dmk\") pod \"payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc\" (UID: \"db1cb268-bac3-4654-9fc2-b3cef57d36de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" Sep 10 19:28:00.750122 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:00.750102 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1cb268-bac3-4654-9fc2-b3cef57d36de-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc\" (UID: \"db1cb268-bac3-4654-9fc2-b3cef57d36de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" Sep 10 19:28:00.766018 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:00.765997 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4dmk\" (UniqueName: \"kubernetes.io/projected/db1cb268-bac3-4654-9fc2-b3cef57d36de-kube-api-access-f4dmk\") pod \"payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc\" (UID: \"db1cb268-bac3-4654-9fc2-b3cef57d36de\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" Sep 10 19:28:01.619178 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:01.619151 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-85e90bf8-dockercfg-nnl6w\"" Sep 10 19:28:01.626290 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:01.626268 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" Sep 10 19:28:01.899279 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:01.899255 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc"] Sep 10 19:28:01.901989 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:28:01.901962 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1cb268_bac3_4654_9fc2_b3cef57d36de.slice/crio-29ec31af520707a14257c178593d83f291c227cc68444cdcd23221247a74c1a0 WatchSource:0}: Error finding container 29ec31af520707a14257c178593d83f291c227cc68444cdcd23221247a74c1a0: Status 404 returned error can't find the container with id 29ec31af520707a14257c178593d83f291c227cc68444cdcd23221247a74c1a0 Sep 10 19:28:02.311079 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:02.310900 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" event={"ID":"db1cb268-bac3-4654-9fc2-b3cef57d36de","Type":"ContainerStarted","Data":"6ddc181fe786de82e4f4e8f98ddd8f187d1355c73149adb29a9a938a5853711a"} Sep 10 19:28:02.311079 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:02.310971 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" Sep 10 19:28:02.311079 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:02.310996 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" event={"ID":"db1cb268-bac3-4654-9fc2-b3cef57d36de","Type":"ContainerStarted","Data":"29ec31af520707a14257c178593d83f291c227cc68444cdcd23221247a74c1a0"} Sep 10 19:28:02.329186 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:02.329137 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" podStartSLOduration=2.32912152 podStartE2EDuration="2.32912152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:02.327534409 +0000 UTC m=+731.458036039" watchObservedRunningTime="2026-09-10 19:28:02.32912152 +0000 UTC m=+731.459623214" Sep 10 19:28:07.286936 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:07.286899 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" Sep 10 19:28:07.287317 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:07.286992 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" Sep 10 19:28:13.318492 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:13.318463 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" Sep 10 19:28:16.931508 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:16.931476 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn"] Sep 10 19:28:16.934828 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:16.934809 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" Sep 10 19:28:16.937390 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:16.937369 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4247c7-dockercfg-xkdlk\"" Sep 10 19:28:16.937641 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:16.937621 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4247c7\"" Sep 10 19:28:16.947193 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:16.947171 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn"] Sep 10 19:28:16.956202 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:16.956175 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl"] Sep 10 19:28:16.959580 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:16.959559 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" Sep 10 19:28:16.969536 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:16.969515 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl"] Sep 10 19:28:17.094127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.094088 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn6zf\" (UniqueName: \"kubernetes.io/projected/7660a028-b331-4459-bbe6-2e0d913bc063-kube-api-access-cn6zf\") pod \"payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl\" (UID: \"7660a028-b331-4459-bbe6-2e0d913bc063\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" Sep 10 19:28:17.094303 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.094132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8671a993-6178-4d76-8a53-8d49ce48bc40-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn\" (UID: \"8671a993-6178-4d76-8a53-8d49ce48bc40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" Sep 10 19:28:17.094303 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.094163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7660a028-b331-4459-bbe6-2e0d913bc063-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl\" (UID: \"7660a028-b331-4459-bbe6-2e0d913bc063\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" Sep 10 19:28:17.094303 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.094260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8xsn\" (UniqueName: \"kubernetes.io/projected/8671a993-6178-4d76-8a53-8d49ce48bc40-kube-api-access-m8xsn\") pod \"payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn\" (UID: \"8671a993-6178-4d76-8a53-8d49ce48bc40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" Sep 10 19:28:17.195352 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.195259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn6zf\" (UniqueName: \"kubernetes.io/projected/7660a028-b331-4459-bbe6-2e0d913bc063-kube-api-access-cn6zf\") pod \"payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl\" (UID: \"7660a028-b331-4459-bbe6-2e0d913bc063\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" Sep 10 19:28:17.195352 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.195297 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8671a993-6178-4d76-8a53-8d49ce48bc40-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn\" (UID: \"8671a993-6178-4d76-8a53-8d49ce48bc40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" Sep 10 19:28:17.195352 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.195318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7660a028-b331-4459-bbe6-2e0d913bc063-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl\" (UID: \"7660a028-b331-4459-bbe6-2e0d913bc063\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" Sep 10 19:28:17.195352 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.195349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8xsn\" (UniqueName: \"kubernetes.io/projected/8671a993-6178-4d76-8a53-8d49ce48bc40-kube-api-access-m8xsn\") pod \"payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn\" (UID: \"8671a993-6178-4d76-8a53-8d49ce48bc40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" Sep 10 19:28:17.196051 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.196029 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8671a993-6178-4d76-8a53-8d49ce48bc40-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn\" (UID: \"8671a993-6178-4d76-8a53-8d49ce48bc40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" Sep 10 19:28:17.196131 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.196063 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7660a028-b331-4459-bbe6-2e0d913bc063-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl\" (UID: \"7660a028-b331-4459-bbe6-2e0d913bc063\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" Sep 10 19:28:17.203930 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.203904 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn6zf\" (UniqueName: \"kubernetes.io/projected/7660a028-b331-4459-bbe6-2e0d913bc063-kube-api-access-cn6zf\") pod \"payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl\" (UID: \"7660a028-b331-4459-bbe6-2e0d913bc063\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" Sep 10 19:28:17.204018 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.203904 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8xsn\" (UniqueName: \"kubernetes.io/projected/8671a993-6178-4d76-8a53-8d49ce48bc40-kube-api-access-m8xsn\") pod \"payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn\" (UID: \"8671a993-6178-4d76-8a53-8d49ce48bc40\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" Sep 10 19:28:17.244805 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.244776 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" Sep 10 19:28:17.271849 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.271749 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" Sep 10 19:28:17.527056 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.526979 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn"] Sep 10 19:28:17.530594 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:28:17.530570 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8671a993_6178_4d76_8a53_8d49ce48bc40.slice/crio-dc64c6f5e621319472859f478b913ffc8819d8af7d05e002e5b8ee7a055f2598 WatchSource:0}: Error finding container dc64c6f5e621319472859f478b913ffc8819d8af7d05e002e5b8ee7a055f2598: Status 404 returned error can't find the container with id dc64c6f5e621319472859f478b913ffc8819d8af7d05e002e5b8ee7a055f2598 Sep 10 19:28:17.561680 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:17.561658 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl"] Sep 10 19:28:17.564046 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:28:17.564021 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7660a028_b331_4459_bbe6_2e0d913bc063.slice/crio-7244c89c1b5cbecf6ac46f35695bcc4d74a0c4f29fa8ef40467ca0fda1b4cf1e WatchSource:0}: Error finding container 7244c89c1b5cbecf6ac46f35695bcc4d74a0c4f29fa8ef40467ca0fda1b4cf1e: Status 404 returned error can't find the container with id 7244c89c1b5cbecf6ac46f35695bcc4d74a0c4f29fa8ef40467ca0fda1b4cf1e Sep 10 19:28:18.387803 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:18.387753 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" event={"ID":"7660a028-b331-4459-bbe6-2e0d913bc063","Type":"ContainerStarted","Data":"e55953b8fc3a7af8752f596e8e06366079a8a13b205c56afcbd2df6a95077fe6"} Sep 10 19:28:18.388373 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:18.388342 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" event={"ID":"7660a028-b331-4459-bbe6-2e0d913bc063","Type":"ContainerStarted","Data":"7244c89c1b5cbecf6ac46f35695bcc4d74a0c4f29fa8ef40467ca0fda1b4cf1e"} Sep 10 19:28:18.388516 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:18.388404 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" Sep 10 19:28:18.389792 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:18.389764 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" event={"ID":"8671a993-6178-4d76-8a53-8d49ce48bc40","Type":"ContainerStarted","Data":"6d1e96e681c4cc57f3ef371ead08a30c3e62bd147b279ee8e910b25655ace1d6"} Sep 10 19:28:18.389921 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:18.389841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" event={"ID":"8671a993-6178-4d76-8a53-8d49ce48bc40","Type":"ContainerStarted","Data":"dc64c6f5e621319472859f478b913ffc8819d8af7d05e002e5b8ee7a055f2598"} Sep 10 19:28:18.389921 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:18.389872 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" Sep 10 19:28:18.405483 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:18.405441 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" podStartSLOduration=2.405426803 podStartE2EDuration="2.405426803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:18.403697567 +0000 UTC m=+747.534199199" watchObservedRunningTime="2026-09-10 19:28:18.405426803 +0000 UTC m=+747.535928428" Sep 10 19:28:18.421463 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:18.421426 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" podStartSLOduration=2.421412522 podStartE2EDuration="2.421412522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:18.420481025 +0000 UTC m=+747.550982656" watchObservedRunningTime="2026-09-10 19:28:18.421412522 +0000 UTC m=+747.551914185" Sep 10 19:28:23.530606 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:23.530573 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7"] Sep 10 19:28:23.535216 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:23.535197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7" Sep 10 19:28:23.537485 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:23.537461 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkgt8\"" Sep 10 19:28:23.537587 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:23.537461 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:28:23.543553 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:23.543529 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7"] Sep 10 19:28:23.650256 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:23.650221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2066eeca-42c9-42a4-a8d1-c68bdd70a47c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7\" (UID: \"2066eeca-42c9-42a4-a8d1-c68bdd70a47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7" Sep 10 19:28:23.750668 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:23.750627 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2066eeca-42c9-42a4-a8d1-c68bdd70a47c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7\" (UID: \"2066eeca-42c9-42a4-a8d1-c68bdd70a47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7" Sep 10 19:28:23.751205 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:23.751185 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2066eeca-42c9-42a4-a8d1-c68bdd70a47c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7\" (UID: \"2066eeca-42c9-42a4-a8d1-c68bdd70a47c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7" Sep 10 19:28:23.847451 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:23.847383 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7" Sep 10 19:28:24.127107 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:24.127024 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7"] Sep 10 19:28:24.129275 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:28:24.129247 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2066eeca_42c9_42a4_a8d1_c68bdd70a47c.slice/crio-1ee534e8c73a78cf113a0495648af7fd8d9d6173092e3ffc585ef26ee5b00c4a WatchSource:0}: Error finding container 1ee534e8c73a78cf113a0495648af7fd8d9d6173092e3ffc585ef26ee5b00c4a: Status 404 returned error can't find the container with id 1ee534e8c73a78cf113a0495648af7fd8d9d6173092e3ffc585ef26ee5b00c4a Sep 10 19:28:24.424529 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:24.424450 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7" event={"ID":"2066eeca-42c9-42a4-a8d1-c68bdd70a47c","Type":"ContainerStarted","Data":"1ee534e8c73a78cf113a0495648af7fd8d9d6173092e3ffc585ef26ee5b00c4a"} Sep 10 19:28:26.435902 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:26.435862 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7" event={"ID":"2066eeca-42c9-42a4-a8d1-c68bdd70a47c","Type":"ContainerStarted","Data":"3c04f7dc460485adb3295f156c081f69b197dfb238a7e578e4a1f8db6b18164e"} Sep 10 19:28:26.480091 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:26.480042 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7" podStartSLOduration=1.365347907 podStartE2EDuration="3.480026375s" totalImagesPullingTime="2.114678468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:26.477085733 +0000 UTC m=+755.607587364" watchObservedRunningTime="2026-09-10 19:28:26.480026375 +0000 UTC m=+755.610528005" Sep 10 19:28:27.441942 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:27.441903 2578 generic.go:356] "Generic (PLEG): container finished" podID="2066eeca-42c9-42a4-a8d1-c68bdd70a47c" containerID="3c04f7dc460485adb3295f156c081f69b197dfb238a7e578e4a1f8db6b18164e" exitCode=0 Sep 10 19:28:27.442324 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:27.441979 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7" event={"ID":"2066eeca-42c9-42a4-a8d1-c68bdd70a47c","Type":"ContainerDied","Data":"3c04f7dc460485adb3295f156c081f69b197dfb238a7e578e4a1f8db6b18164e"} Sep 10 19:28:28.577908 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:28.577885 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7" Sep 10 19:28:28.698771 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:28.698689 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2066eeca-42c9-42a4-a8d1-c68bdd70a47c-maas-api-service-ca\") pod \"2066eeca-42c9-42a4-a8d1-c68bdd70a47c\" (UID: \"2066eeca-42c9-42a4-a8d1-c68bdd70a47c\") " Sep 10 19:28:28.699047 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:28.699023 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2066eeca-42c9-42a4-a8d1-c68bdd70a47c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2066eeca-42c9-42a4-a8d1-c68bdd70a47c" (UID: "2066eeca-42c9-42a4-a8d1-c68bdd70a47c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:28:28.799772 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:28.799716 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2066eeca-42c9-42a4-a8d1-c68bdd70a47c-maas-api-service-ca\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:28:29.403810 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:29.403784 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" Sep 10 19:28:29.453294 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:29.453216 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7" Sep 10 19:28:29.453294 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:29.453248 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7" event={"ID":"2066eeca-42c9-42a4-a8d1-c68bdd70a47c","Type":"ContainerDied","Data":"1ee534e8c73a78cf113a0495648af7fd8d9d6173092e3ffc585ef26ee5b00c4a"} Sep 10 19:28:29.453453 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:29.453301 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ee534e8c73a78cf113a0495648af7fd8d9d6173092e3ffc585ef26ee5b00c4a" Sep 10 19:28:30.408687 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:30.408655 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" Sep 10 19:28:40.462188 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:40.462155 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc"] Sep 10 19:28:40.462753 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:40.462570 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" podUID="db1cb268-bac3-4654-9fc2-b3cef57d36de" containerName="payload-pre-processing" containerID="cri-o://6ddc181fe786de82e4f4e8f98ddd8f187d1355c73149adb29a9a938a5853711a" gracePeriod=30 Sep 10 19:28:40.663078 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:40.663048 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7"] Sep 10 19:28:40.668667 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:40.668631 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-85e90bf8-d7wh7"] Sep 10 19:28:40.727828 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:40.727802 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" Sep 10 19:28:40.799764 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:40.799045 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1cb268-bac3-4654-9fc2-b3cef57d36de-plugins-config-volume\") pod \"db1cb268-bac3-4654-9fc2-b3cef57d36de\" (UID: \"db1cb268-bac3-4654-9fc2-b3cef57d36de\") " Sep 10 19:28:40.799764 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:40.799151 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f4dmk\" (UniqueName: \"kubernetes.io/projected/db1cb268-bac3-4654-9fc2-b3cef57d36de-kube-api-access-f4dmk\") pod \"db1cb268-bac3-4654-9fc2-b3cef57d36de\" (UID: \"db1cb268-bac3-4654-9fc2-b3cef57d36de\") " Sep 10 19:28:40.799764 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:40.799510 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db1cb268-bac3-4654-9fc2-b3cef57d36de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db1cb268-bac3-4654-9fc2-b3cef57d36de" (UID: "db1cb268-bac3-4654-9fc2-b3cef57d36de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:28:40.802937 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:40.802910 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1cb268-bac3-4654-9fc2-b3cef57d36de-kube-api-access-f4dmk" (OuterVolumeSpecName: "kube-api-access-f4dmk") pod "db1cb268-bac3-4654-9fc2-b3cef57d36de" (UID: "db1cb268-bac3-4654-9fc2-b3cef57d36de"). InnerVolumeSpecName "kube-api-access-f4dmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:28:40.900866 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:40.900830 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4dmk\" (UniqueName: \"kubernetes.io/projected/db1cb268-bac3-4654-9fc2-b3cef57d36de-kube-api-access-f4dmk\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:28:40.900866 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:40.900865 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db1cb268-bac3-4654-9fc2-b3cef57d36de-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:28:41.390371 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:41.390337 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2066eeca-42c9-42a4-a8d1-c68bdd70a47c" path="/var/lib/kubelet/pods/2066eeca-42c9-42a4-a8d1-c68bdd70a47c/volumes" Sep 10 19:28:41.508503 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:41.508467 2578 generic.go:356] "Generic (PLEG): container finished" podID="db1cb268-bac3-4654-9fc2-b3cef57d36de" containerID="6ddc181fe786de82e4f4e8f98ddd8f187d1355c73149adb29a9a938a5853711a" exitCode=0 Sep 10 19:28:41.508873 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:41.508521 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" Sep 10 19:28:41.508873 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:41.508536 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" event={"ID":"db1cb268-bac3-4654-9fc2-b3cef57d36de","Type":"ContainerDied","Data":"6ddc181fe786de82e4f4e8f98ddd8f187d1355c73149adb29a9a938a5853711a"} Sep 10 19:28:41.508873 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:41.508576 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc" event={"ID":"db1cb268-bac3-4654-9fc2-b3cef57d36de","Type":"ContainerDied","Data":"29ec31af520707a14257c178593d83f291c227cc68444cdcd23221247a74c1a0"} Sep 10 19:28:41.508873 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:41.508598 2578 scope.go:117] "RemoveContainer" containerID="6ddc181fe786de82e4f4e8f98ddd8f187d1355c73149adb29a9a938a5853711a" Sep 10 19:28:41.517595 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:41.517576 2578 scope.go:117] "RemoveContainer" containerID="6ddc181fe786de82e4f4e8f98ddd8f187d1355c73149adb29a9a938a5853711a" Sep 10 19:28:41.517959 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:28:41.517939 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ddc181fe786de82e4f4e8f98ddd8f187d1355c73149adb29a9a938a5853711a\": container with ID starting with 6ddc181fe786de82e4f4e8f98ddd8f187d1355c73149adb29a9a938a5853711a not found: ID does not exist" containerID="6ddc181fe786de82e4f4e8f98ddd8f187d1355c73149adb29a9a938a5853711a" Sep 10 19:28:41.518023 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:41.517967 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ddc181fe786de82e4f4e8f98ddd8f187d1355c73149adb29a9a938a5853711a"} err="failed to get container status \"6ddc181fe786de82e4f4e8f98ddd8f187d1355c73149adb29a9a938a5853711a\": rpc error: code = NotFound desc = could not find container \"6ddc181fe786de82e4f4e8f98ddd8f187d1355c73149adb29a9a938a5853711a\": container with ID starting with 6ddc181fe786de82e4f4e8f98ddd8f187d1355c73149adb29a9a938a5853711a not found: ID does not exist" Sep 10 19:28:41.530243 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:41.530220 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc"] Sep 10 19:28:41.539991 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:41.539971 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-85e90bf8-5755c99898-75fjc"] Sep 10 19:28:43.390855 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:43.390822 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db1cb268-bac3-4654-9fc2-b3cef57d36de" path="/var/lib/kubelet/pods/db1cb268-bac3-4654-9fc2-b3cef57d36de/volumes" Sep 10 19:28:45.391568 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.391534 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq"] Sep 10 19:28:45.391934 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.391872 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1cb268-bac3-4654-9fc2-b3cef57d36de" containerName="payload-pre-processing" Sep 10 19:28:45.391934 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.391883 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1cb268-bac3-4654-9fc2-b3cef57d36de" containerName="payload-pre-processing" Sep 10 19:28:45.391934 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.391893 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2066eeca-42c9-42a4-a8d1-c68bdd70a47c" containerName="revoke-keys" Sep 10 19:28:45.391934 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.391899 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2066eeca-42c9-42a4-a8d1-c68bdd70a47c" containerName="revoke-keys" Sep 10 19:28:45.392058 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.391955 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2066eeca-42c9-42a4-a8d1-c68bdd70a47c" containerName="revoke-keys" Sep 10 19:28:45.392058 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.391963 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="db1cb268-bac3-4654-9fc2-b3cef57d36de" containerName="payload-pre-processing" Sep 10 19:28:45.394717 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.394701 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" Sep 10 19:28:45.396858 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.396829 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-80c77821-dockercfg-blc86\"" Sep 10 19:28:45.396982 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.396835 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-80c77821\"" Sep 10 19:28:45.404251 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.404228 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq"] Sep 10 19:28:45.421106 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.421084 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t"] Sep 10 19:28:45.424300 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.424268 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" Sep 10 19:28:45.434865 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.434846 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t"] Sep 10 19:28:45.440218 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.440195 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a12706d-cb4d-4a84-8366-a53eed4dd525-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq\" (UID: \"9a12706d-cb4d-4a84-8366-a53eed4dd525\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" Sep 10 19:28:45.440332 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.440224 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2glw\" (UniqueName: \"kubernetes.io/projected/9a12706d-cb4d-4a84-8366-a53eed4dd525-kube-api-access-n2glw\") pod \"payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq\" (UID: \"9a12706d-cb4d-4a84-8366-a53eed4dd525\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" Sep 10 19:28:45.541534 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.541498 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aabd3a96-4ec3-438c-9578-8ded1ee3e462-plugins-config-volume\") pod \"payload-processing-e2e-migrate-80c77821-774f544854-wf28t\" (UID: \"aabd3a96-4ec3-438c-9578-8ded1ee3e462\") " pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" Sep 10 19:28:45.541695 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.541551 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a12706d-cb4d-4a84-8366-a53eed4dd525-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq\" (UID: \"9a12706d-cb4d-4a84-8366-a53eed4dd525\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" Sep 10 19:28:45.541695 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.541570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2glw\" (UniqueName: \"kubernetes.io/projected/9a12706d-cb4d-4a84-8366-a53eed4dd525-kube-api-access-n2glw\") pod \"payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq\" (UID: \"9a12706d-cb4d-4a84-8366-a53eed4dd525\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" Sep 10 19:28:45.541695 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.541602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk4k4\" (UniqueName: \"kubernetes.io/projected/aabd3a96-4ec3-438c-9578-8ded1ee3e462-kube-api-access-wk4k4\") pod \"payload-processing-e2e-migrate-80c77821-774f544854-wf28t\" (UID: \"aabd3a96-4ec3-438c-9578-8ded1ee3e462\") " pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" Sep 10 19:28:45.542181 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.542163 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a12706d-cb4d-4a84-8366-a53eed4dd525-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq\" (UID: \"9a12706d-cb4d-4a84-8366-a53eed4dd525\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" Sep 10 19:28:45.550753 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.550710 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2glw\" (UniqueName: \"kubernetes.io/projected/9a12706d-cb4d-4a84-8366-a53eed4dd525-kube-api-access-n2glw\") pod \"payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq\" (UID: \"9a12706d-cb4d-4a84-8366-a53eed4dd525\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" Sep 10 19:28:45.642063 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.641997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aabd3a96-4ec3-438c-9578-8ded1ee3e462-plugins-config-volume\") pod \"payload-processing-e2e-migrate-80c77821-774f544854-wf28t\" (UID: \"aabd3a96-4ec3-438c-9578-8ded1ee3e462\") " pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" Sep 10 19:28:45.642370 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.642062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk4k4\" (UniqueName: \"kubernetes.io/projected/aabd3a96-4ec3-438c-9578-8ded1ee3e462-kube-api-access-wk4k4\") pod \"payload-processing-e2e-migrate-80c77821-774f544854-wf28t\" (UID: \"aabd3a96-4ec3-438c-9578-8ded1ee3e462\") " pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" Sep 10 19:28:45.642614 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.642594 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aabd3a96-4ec3-438c-9578-8ded1ee3e462-plugins-config-volume\") pod \"payload-processing-e2e-migrate-80c77821-774f544854-wf28t\" (UID: \"aabd3a96-4ec3-438c-9578-8ded1ee3e462\") " pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" Sep 10 19:28:45.650994 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.650968 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk4k4\" (UniqueName: \"kubernetes.io/projected/aabd3a96-4ec3-438c-9578-8ded1ee3e462-kube-api-access-wk4k4\") pod \"payload-processing-e2e-migrate-80c77821-774f544854-wf28t\" (UID: \"aabd3a96-4ec3-438c-9578-8ded1ee3e462\") " pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" Sep 10 19:28:45.705806 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.705782 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" Sep 10 19:28:45.734516 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.734487 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" Sep 10 19:28:45.979421 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:45.979382 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq"] Sep 10 19:28:45.981804 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:28:45.981773 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a12706d_cb4d_4a84_8366_a53eed4dd525.slice/crio-72424cf6769526c56807b523b0402e5dcba8b3d9aa153c426749b9cf8a93fa7b WatchSource:0}: Error finding container 72424cf6769526c56807b523b0402e5dcba8b3d9aa153c426749b9cf8a93fa7b: Status 404 returned error can't find the container with id 72424cf6769526c56807b523b0402e5dcba8b3d9aa153c426749b9cf8a93fa7b Sep 10 19:28:46.016836 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:46.016814 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t"] Sep 10 19:28:46.018217 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:28:46.018198 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaabd3a96_4ec3_438c_9578_8ded1ee3e462.slice/crio-3437694683a0320cbed12d45867d8859b5aaae35f59d98310b356d242dcf6abc WatchSource:0}: Error finding container 3437694683a0320cbed12d45867d8859b5aaae35f59d98310b356d242dcf6abc: Status 404 returned error can't find the container with id 3437694683a0320cbed12d45867d8859b5aaae35f59d98310b356d242dcf6abc Sep 10 19:28:46.535252 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:46.535214 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" event={"ID":"aabd3a96-4ec3-438c-9578-8ded1ee3e462","Type":"ContainerStarted","Data":"e7bc6dbf27b5ea48caaa64b636690b0a5c7d868bd95c12427cc77a86293fb5c4"} Sep 10 19:28:46.535708 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:46.535290 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" Sep 10 19:28:46.535708 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:46.535316 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" event={"ID":"aabd3a96-4ec3-438c-9578-8ded1ee3e462","Type":"ContainerStarted","Data":"3437694683a0320cbed12d45867d8859b5aaae35f59d98310b356d242dcf6abc"} Sep 10 19:28:46.536844 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:46.536822 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" event={"ID":"9a12706d-cb4d-4a84-8366-a53eed4dd525","Type":"ContainerStarted","Data":"a2bdab89a911f8579fcbb4254525ff6057897381dbaf1236666f0d767a12917f"} Sep 10 19:28:46.536961 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:46.536867 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" event={"ID":"9a12706d-cb4d-4a84-8366-a53eed4dd525","Type":"ContainerStarted","Data":"72424cf6769526c56807b523b0402e5dcba8b3d9aa153c426749b9cf8a93fa7b"} Sep 10 19:28:46.536961 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:46.536915 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" Sep 10 19:28:46.553289 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:46.553233 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" podStartSLOduration=1.553207847 podStartE2EDuration="1.553207847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:46.550229344 +0000 UTC m=+775.680730975" watchObservedRunningTime="2026-09-10 19:28:46.553207847 +0000 UTC m=+775.683709481" Sep 10 19:28:46.573686 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:46.573644 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" podStartSLOduration=1.573629387 podStartE2EDuration="1.573629387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:28:46.572054693 +0000 UTC m=+775.702556316" watchObservedRunningTime="2026-09-10 19:28:46.573629387 +0000 UTC m=+775.704131071" Sep 10 19:28:57.544092 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:57.544059 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" Sep 10 19:28:57.544473 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:28:57.544240 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" Sep 10 19:29:09.359279 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:09.359245 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq"] Sep 10 19:29:09.365949 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:09.365931 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq" Sep 10 19:29:09.368685 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:09.368665 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:29:09.368914 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:09.368894 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkgt8\"" Sep 10 19:29:09.373874 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:09.373850 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq"] Sep 10 19:29:09.455720 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:09.455692 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e30852-5b93-4553-a585-8073ba1bac49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq\" (UID: \"30e30852-5b93-4553-a585-8073ba1bac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq" Sep 10 19:29:09.556717 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:09.556681 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e30852-5b93-4553-a585-8073ba1bac49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq\" (UID: \"30e30852-5b93-4553-a585-8073ba1bac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq" Sep 10 19:29:09.557390 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:09.557370 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e30852-5b93-4553-a585-8073ba1bac49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq\" (UID: \"30e30852-5b93-4553-a585-8073ba1bac49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq" Sep 10 19:29:09.677114 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:09.677084 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq" Sep 10 19:29:09.952352 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:09.952327 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq"] Sep 10 19:29:09.954076 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:29:09.954046 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e30852_5b93_4553_a585_8073ba1bac49.slice/crio-73607a12aeea8c94301a060f8c0a34bcc00a450a4d227e7373c82927c2e3f25c WatchSource:0}: Error finding container 73607a12aeea8c94301a060f8c0a34bcc00a450a4d227e7373c82927c2e3f25c: Status 404 returned error can't find the container with id 73607a12aeea8c94301a060f8c0a34bcc00a450a4d227e7373c82927c2e3f25c Sep 10 19:29:10.649618 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:10.649584 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq" event={"ID":"30e30852-5b93-4553-a585-8073ba1bac49","Type":"ContainerStarted","Data":"1c3439e01dd2fb914d1173beba4f9ef7b30aa3dbe5c2e99213d00a96ca954658"} Sep 10 19:29:10.650004 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:10.649638 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq" event={"ID":"30e30852-5b93-4553-a585-8073ba1bac49","Type":"ContainerStarted","Data":"73607a12aeea8c94301a060f8c0a34bcc00a450a4d227e7373c82927c2e3f25c"} Sep 10 19:29:10.665277 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:10.665237 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq" podStartSLOduration=1.665223168 podStartE2EDuration="1.665223168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:29:10.664056777 +0000 UTC m=+799.794558409" watchObservedRunningTime="2026-09-10 19:29:10.665223168 +0000 UTC m=+799.795724832" Sep 10 19:29:11.661491 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:11.661454 2578 generic.go:356] "Generic (PLEG): container finished" podID="30e30852-5b93-4553-a585-8073ba1bac49" containerID="1c3439e01dd2fb914d1173beba4f9ef7b30aa3dbe5c2e99213d00a96ca954658" exitCode=0 Sep 10 19:29:11.661886 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:11.661536 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq" event={"ID":"30e30852-5b93-4553-a585-8073ba1bac49","Type":"ContainerDied","Data":"1c3439e01dd2fb914d1173beba4f9ef7b30aa3dbe5c2e99213d00a96ca954658"} Sep 10 19:29:12.806554 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:12.806532 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq" Sep 10 19:29:12.990859 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:12.990782 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e30852-5b93-4553-a585-8073ba1bac49-maas-api-service-ca\") pod \"30e30852-5b93-4553-a585-8073ba1bac49\" (UID: \"30e30852-5b93-4553-a585-8073ba1bac49\") " Sep 10 19:29:12.991137 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:12.991116 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e30852-5b93-4553-a585-8073ba1bac49-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30e30852-5b93-4553-a585-8073ba1bac49" (UID: "30e30852-5b93-4553-a585-8073ba1bac49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:29:13.091932 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:13.091898 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30e30852-5b93-4553-a585-8073ba1bac49-maas-api-service-ca\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:29:13.672845 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:13.672759 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq" Sep 10 19:29:13.672845 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:13.672778 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq" event={"ID":"30e30852-5b93-4553-a585-8073ba1bac49","Type":"ContainerDied","Data":"73607a12aeea8c94301a060f8c0a34bcc00a450a4d227e7373c82927c2e3f25c"} Sep 10 19:29:13.672845 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:13.672832 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73607a12aeea8c94301a060f8c0a34bcc00a450a4d227e7373c82927c2e3f25c" Sep 10 19:29:16.106985 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.106951 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t"] Sep 10 19:29:16.107438 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.107208 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" podUID="aabd3a96-4ec3-438c-9578-8ded1ee3e462" containerName="payload-processing" containerID="cri-o://e7bc6dbf27b5ea48caaa64b636690b0a5c7d868bd95c12427cc77a86293fb5c4" gracePeriod=30 Sep 10 19:29:16.114846 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.114820 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq"] Sep 10 19:29:16.115055 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.115029 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" podUID="9a12706d-cb4d-4a84-8366-a53eed4dd525" containerName="payload-pre-processing" containerID="cri-o://a2bdab89a911f8579fcbb4254525ff6057897381dbaf1236666f0d767a12917f" gracePeriod=30 Sep 10 19:29:16.292435 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.292366 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq"] Sep 10 19:29:16.294111 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.294090 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-80c77821-t9jxq"] Sep 10 19:29:16.391578 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.391547 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" Sep 10 19:29:16.395613 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.395343 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" Sep 10 19:29:16.418606 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.418568 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aabd3a96-4ec3-438c-9578-8ded1ee3e462-plugins-config-volume\") pod \"aabd3a96-4ec3-438c-9578-8ded1ee3e462\" (UID: \"aabd3a96-4ec3-438c-9578-8ded1ee3e462\") " Sep 10 19:29:16.418777 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.418627 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2glw\" (UniqueName: \"kubernetes.io/projected/9a12706d-cb4d-4a84-8366-a53eed4dd525-kube-api-access-n2glw\") pod \"9a12706d-cb4d-4a84-8366-a53eed4dd525\" (UID: \"9a12706d-cb4d-4a84-8366-a53eed4dd525\") " Sep 10 19:29:16.418777 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.418676 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a12706d-cb4d-4a84-8366-a53eed4dd525-plugins-config-volume\") pod \"9a12706d-cb4d-4a84-8366-a53eed4dd525\" (UID: \"9a12706d-cb4d-4a84-8366-a53eed4dd525\") " Sep 10 19:29:16.418777 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.418710 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wk4k4\" (UniqueName: \"kubernetes.io/projected/aabd3a96-4ec3-438c-9578-8ded1ee3e462-kube-api-access-wk4k4\") pod \"aabd3a96-4ec3-438c-9578-8ded1ee3e462\" (UID: \"aabd3a96-4ec3-438c-9578-8ded1ee3e462\") " Sep 10 19:29:16.419164 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.419077 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a12706d-cb4d-4a84-8366-a53eed4dd525-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a12706d-cb4d-4a84-8366-a53eed4dd525" (UID: "9a12706d-cb4d-4a84-8366-a53eed4dd525"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:29:16.419578 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.419555 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aabd3a96-4ec3-438c-9578-8ded1ee3e462-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aabd3a96-4ec3-438c-9578-8ded1ee3e462" (UID: "aabd3a96-4ec3-438c-9578-8ded1ee3e462"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:29:16.421767 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.421714 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aabd3a96-4ec3-438c-9578-8ded1ee3e462-kube-api-access-wk4k4" (OuterVolumeSpecName: "kube-api-access-wk4k4") pod "aabd3a96-4ec3-438c-9578-8ded1ee3e462" (UID: "aabd3a96-4ec3-438c-9578-8ded1ee3e462"). InnerVolumeSpecName "kube-api-access-wk4k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:29:16.422311 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.422286 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a12706d-cb4d-4a84-8366-a53eed4dd525-kube-api-access-n2glw" (OuterVolumeSpecName: "kube-api-access-n2glw") pod "9a12706d-cb4d-4a84-8366-a53eed4dd525" (UID: "9a12706d-cb4d-4a84-8366-a53eed4dd525"). InnerVolumeSpecName "kube-api-access-n2glw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:29:16.519865 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.519833 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a12706d-cb4d-4a84-8366-a53eed4dd525-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:29:16.519865 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.519858 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk4k4\" (UniqueName: \"kubernetes.io/projected/aabd3a96-4ec3-438c-9578-8ded1ee3e462-kube-api-access-wk4k4\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:29:16.519865 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.519866 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aabd3a96-4ec3-438c-9578-8ded1ee3e462-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:29:16.519865 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.519875 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2glw\" (UniqueName: \"kubernetes.io/projected/9a12706d-cb4d-4a84-8366-a53eed4dd525-kube-api-access-n2glw\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:29:16.689268 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.689238 2578 generic.go:356] "Generic (PLEG): container finished" podID="aabd3a96-4ec3-438c-9578-8ded1ee3e462" containerID="e7bc6dbf27b5ea48caaa64b636690b0a5c7d868bd95c12427cc77a86293fb5c4" exitCode=0 Sep 10 19:29:16.689433 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.689318 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" Sep 10 19:29:16.689433 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.689330 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" event={"ID":"aabd3a96-4ec3-438c-9578-8ded1ee3e462","Type":"ContainerDied","Data":"e7bc6dbf27b5ea48caaa64b636690b0a5c7d868bd95c12427cc77a86293fb5c4"} Sep 10 19:29:16.689535 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.689430 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t" event={"ID":"aabd3a96-4ec3-438c-9578-8ded1ee3e462","Type":"ContainerDied","Data":"3437694683a0320cbed12d45867d8859b5aaae35f59d98310b356d242dcf6abc"} Sep 10 19:29:16.689535 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.689465 2578 scope.go:117] "RemoveContainer" containerID="e7bc6dbf27b5ea48caaa64b636690b0a5c7d868bd95c12427cc77a86293fb5c4" Sep 10 19:29:16.690822 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.690796 2578 generic.go:356] "Generic (PLEG): container finished" podID="9a12706d-cb4d-4a84-8366-a53eed4dd525" containerID="a2bdab89a911f8579fcbb4254525ff6057897381dbaf1236666f0d767a12917f" exitCode=0 Sep 10 19:29:16.690956 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.690867 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" Sep 10 19:29:16.690956 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.690885 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" event={"ID":"9a12706d-cb4d-4a84-8366-a53eed4dd525","Type":"ContainerDied","Data":"a2bdab89a911f8579fcbb4254525ff6057897381dbaf1236666f0d767a12917f"} Sep 10 19:29:16.690956 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.690925 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq" event={"ID":"9a12706d-cb4d-4a84-8366-a53eed4dd525","Type":"ContainerDied","Data":"72424cf6769526c56807b523b0402e5dcba8b3d9aa153c426749b9cf8a93fa7b"} Sep 10 19:29:16.700641 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.700623 2578 scope.go:117] "RemoveContainer" containerID="e7bc6dbf27b5ea48caaa64b636690b0a5c7d868bd95c12427cc77a86293fb5c4" Sep 10 19:29:16.700967 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:29:16.700947 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7bc6dbf27b5ea48caaa64b636690b0a5c7d868bd95c12427cc77a86293fb5c4\": container with ID starting with e7bc6dbf27b5ea48caaa64b636690b0a5c7d868bd95c12427cc77a86293fb5c4 not found: ID does not exist" containerID="e7bc6dbf27b5ea48caaa64b636690b0a5c7d868bd95c12427cc77a86293fb5c4" Sep 10 19:29:16.701046 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.700971 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7bc6dbf27b5ea48caaa64b636690b0a5c7d868bd95c12427cc77a86293fb5c4"} err="failed to get container status \"e7bc6dbf27b5ea48caaa64b636690b0a5c7d868bd95c12427cc77a86293fb5c4\": rpc error: code = NotFound desc = could not find container \"e7bc6dbf27b5ea48caaa64b636690b0a5c7d868bd95c12427cc77a86293fb5c4\": container with ID starting with e7bc6dbf27b5ea48caaa64b636690b0a5c7d868bd95c12427cc77a86293fb5c4 not found: ID does not exist" Sep 10 19:29:16.701046 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.700987 2578 scope.go:117] "RemoveContainer" containerID="a2bdab89a911f8579fcbb4254525ff6057897381dbaf1236666f0d767a12917f" Sep 10 19:29:16.708985 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.708967 2578 scope.go:117] "RemoveContainer" containerID="a2bdab89a911f8579fcbb4254525ff6057897381dbaf1236666f0d767a12917f" Sep 10 19:29:16.709233 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:29:16.709215 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2bdab89a911f8579fcbb4254525ff6057897381dbaf1236666f0d767a12917f\": container with ID starting with a2bdab89a911f8579fcbb4254525ff6057897381dbaf1236666f0d767a12917f not found: ID does not exist" containerID="a2bdab89a911f8579fcbb4254525ff6057897381dbaf1236666f0d767a12917f" Sep 10 19:29:16.709294 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.709243 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2bdab89a911f8579fcbb4254525ff6057897381dbaf1236666f0d767a12917f"} err="failed to get container status \"a2bdab89a911f8579fcbb4254525ff6057897381dbaf1236666f0d767a12917f\": rpc error: code = NotFound desc = could not find container \"a2bdab89a911f8579fcbb4254525ff6057897381dbaf1236666f0d767a12917f\": container with ID starting with a2bdab89a911f8579fcbb4254525ff6057897381dbaf1236666f0d767a12917f not found: ID does not exist" Sep 10 19:29:16.712655 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.712635 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t"] Sep 10 19:29:16.716871 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.716854 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-80c77821-774f544854-wf28t"] Sep 10 19:29:16.727271 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.727253 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq"] Sep 10 19:29:16.733216 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:16.733197 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-80c77821-6679bf5b58-v7qsq"] Sep 10 19:29:17.392296 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:17.392262 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30e30852-5b93-4553-a585-8073ba1bac49" path="/var/lib/kubelet/pods/30e30852-5b93-4553-a585-8073ba1bac49/volumes" Sep 10 19:29:17.392687 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:17.392619 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a12706d-cb4d-4a84-8366-a53eed4dd525" path="/var/lib/kubelet/pods/9a12706d-cb4d-4a84-8366-a53eed4dd525/volumes" Sep 10 19:29:17.392972 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:17.392956 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aabd3a96-4ec3-438c-9578-8ded1ee3e462" path="/var/lib/kubelet/pods/aabd3a96-4ec3-438c-9578-8ded1ee3e462/volumes" Sep 10 19:29:21.345211 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.345182 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww"] Sep 10 19:29:21.345562 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.345527 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a12706d-cb4d-4a84-8366-a53eed4dd525" containerName="payload-pre-processing" Sep 10 19:29:21.345562 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.345538 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a12706d-cb4d-4a84-8366-a53eed4dd525" containerName="payload-pre-processing" Sep 10 19:29:21.345562 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.345548 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aabd3a96-4ec3-438c-9578-8ded1ee3e462" containerName="payload-processing" Sep 10 19:29:21.345562 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.345554 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aabd3a96-4ec3-438c-9578-8ded1ee3e462" containerName="payload-processing" Sep 10 19:29:21.345705 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.345574 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30e30852-5b93-4553-a585-8073ba1bac49" containerName="revoke-keys" Sep 10 19:29:21.345705 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.345580 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30e30852-5b93-4553-a585-8073ba1bac49" containerName="revoke-keys" Sep 10 19:29:21.345705 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.345628 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aabd3a96-4ec3-438c-9578-8ded1ee3e462" containerName="payload-processing" Sep 10 19:29:21.345705 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.345637 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a12706d-cb4d-4a84-8366-a53eed4dd525" containerName="payload-pre-processing" Sep 10 19:29:21.345705 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.345644 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30e30852-5b93-4553-a585-8073ba1bac49" containerName="revoke-keys" Sep 10 19:29:21.350124 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.350103 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" Sep 10 19:29:21.352346 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.352326 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f0b1e2fe\"" Sep 10 19:29:21.360191 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.360166 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww"] Sep 10 19:29:21.362014 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.361996 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn"] Sep 10 19:29:21.365313 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.365295 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" Sep 10 19:29:21.367505 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.367484 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f0b1e2fe-dockercfg-m8t9k\"" Sep 10 19:29:21.376109 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.376088 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn"] Sep 10 19:29:21.466742 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.466695 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb75273-a0fd-40fb-b8c1-33ac13c7c661-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww\" (UID: \"9cb75273-a0fd-40fb-b8c1-33ac13c7c661\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" Sep 10 19:29:21.466917 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.466753 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a-plugins-config-volume\") pod \"payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn\" (UID: \"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a\") " pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" Sep 10 19:29:21.466917 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.466780 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr7rj\" (UniqueName: \"kubernetes.io/projected/9cb75273-a0fd-40fb-b8c1-33ac13c7c661-kube-api-access-nr7rj\") pod \"payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww\" (UID: \"9cb75273-a0fd-40fb-b8c1-33ac13c7c661\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" Sep 10 19:29:21.466917 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.466836 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brwqv\" (UniqueName: \"kubernetes.io/projected/cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a-kube-api-access-brwqv\") pod \"payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn\" (UID: \"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a\") " pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" Sep 10 19:29:21.568148 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.568116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brwqv\" (UniqueName: \"kubernetes.io/projected/cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a-kube-api-access-brwqv\") pod \"payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn\" (UID: \"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a\") " pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" Sep 10 19:29:21.568295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.568204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb75273-a0fd-40fb-b8c1-33ac13c7c661-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww\" (UID: \"9cb75273-a0fd-40fb-b8c1-33ac13c7c661\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" Sep 10 19:29:21.568295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.568232 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a-plugins-config-volume\") pod \"payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn\" (UID: \"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a\") " pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" Sep 10 19:29:21.568295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.568249 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr7rj\" (UniqueName: \"kubernetes.io/projected/9cb75273-a0fd-40fb-b8c1-33ac13c7c661-kube-api-access-nr7rj\") pod \"payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww\" (UID: \"9cb75273-a0fd-40fb-b8c1-33ac13c7c661\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" Sep 10 19:29:21.569166 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.569144 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a-plugins-config-volume\") pod \"payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn\" (UID: \"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a\") " pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" Sep 10 19:29:21.569233 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.569144 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb75273-a0fd-40fb-b8c1-33ac13c7c661-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww\" (UID: \"9cb75273-a0fd-40fb-b8c1-33ac13c7c661\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" Sep 10 19:29:21.577933 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.577898 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brwqv\" (UniqueName: \"kubernetes.io/projected/cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a-kube-api-access-brwqv\") pod \"payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn\" (UID: \"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a\") " pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" Sep 10 19:29:21.578048 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.577991 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr7rj\" (UniqueName: \"kubernetes.io/projected/9cb75273-a0fd-40fb-b8c1-33ac13c7c661-kube-api-access-nr7rj\") pod \"payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww\" (UID: \"9cb75273-a0fd-40fb-b8c1-33ac13c7c661\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" Sep 10 19:29:21.661604 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.661575 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" Sep 10 19:29:21.675249 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.675219 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" Sep 10 19:29:21.941635 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.941612 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww"] Sep 10 19:29:21.943266 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:29:21.943233 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb75273_a0fd_40fb_b8c1_33ac13c7c661.slice/crio-98f710f24bb5ce758ec6cb85c94fcf64f2ff24f2f4a781dcfdc2febecc2c227d WatchSource:0}: Error finding container 98f710f24bb5ce758ec6cb85c94fcf64f2ff24f2f4a781dcfdc2febecc2c227d: Status 404 returned error can't find the container with id 98f710f24bb5ce758ec6cb85c94fcf64f2ff24f2f4a781dcfdc2febecc2c227d Sep 10 19:29:21.965322 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:21.965302 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn"] Sep 10 19:29:21.972700 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:29:21.972674 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2e1cf5_03ba_4a8a_b7e2_56819ba30a0a.slice/crio-a136e0db0862f39039130407e23cdc7efded1d90ee62fb8eba3d2fd254598f95 WatchSource:0}: Error finding container a136e0db0862f39039130407e23cdc7efded1d90ee62fb8eba3d2fd254598f95: Status 404 returned error can't find the container with id a136e0db0862f39039130407e23cdc7efded1d90ee62fb8eba3d2fd254598f95 Sep 10 19:29:22.727894 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:22.727854 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" event={"ID":"9cb75273-a0fd-40fb-b8c1-33ac13c7c661","Type":"ContainerStarted","Data":"181b4685c8660f92f986777ab8335d3af5e5370cd479b6c6e0ec9bac78d67dd0"} Sep 10 19:29:22.728371 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:22.727923 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" event={"ID":"9cb75273-a0fd-40fb-b8c1-33ac13c7c661","Type":"ContainerStarted","Data":"98f710f24bb5ce758ec6cb85c94fcf64f2ff24f2f4a781dcfdc2febecc2c227d"} Sep 10 19:29:22.728371 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:22.728059 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" Sep 10 19:29:22.729553 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:22.729530 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" event={"ID":"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a","Type":"ContainerStarted","Data":"87811b8221e35949204c980009ad1f8a5381bbe8ab68583a881ba12eeb45ec0d"} Sep 10 19:29:22.729644 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:22.729573 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" event={"ID":"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a","Type":"ContainerStarted","Data":"a136e0db0862f39039130407e23cdc7efded1d90ee62fb8eba3d2fd254598f95"} Sep 10 19:29:22.729644 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:22.729603 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" Sep 10 19:29:22.746404 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:22.746359 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" podStartSLOduration=1.746346618 podStartE2EDuration="1.746346618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:29:22.744883496 +0000 UTC m=+811.875385127" watchObservedRunningTime="2026-09-10 19:29:22.746346618 +0000 UTC m=+811.876848285" Sep 10 19:29:22.760494 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:22.760458 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" podStartSLOduration=1.760445704 podStartE2EDuration="1.760445704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:29:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:29:22.758967002 +0000 UTC m=+811.889468643" watchObservedRunningTime="2026-09-10 19:29:22.760445704 +0000 UTC m=+811.890947336" Sep 10 19:29:33.736677 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:33.736642 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" Sep 10 19:29:33.737070 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:33.736711 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" Sep 10 19:29:45.165024 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:45.164986 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql"] Sep 10 19:29:45.168616 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:45.168596 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql" Sep 10 19:29:45.170924 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:45.170897 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:29:45.171045 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:45.170948 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkgt8\"" Sep 10 19:29:45.175525 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:45.175493 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql"] Sep 10 19:29:45.270049 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:45.270008 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql\" (UID: \"e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql" Sep 10 19:29:45.371141 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:45.371108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql\" (UID: \"e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql" Sep 10 19:29:45.371977 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:45.371951 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql\" (UID: \"e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql" Sep 10 19:29:45.482274 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:45.482129 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql" Sep 10 19:29:45.762011 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:45.761658 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql"] Sep 10 19:29:45.764140 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:29:45.764115 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cc24b4_a345_4501_b8a7_6e5a6c80ddcb.slice/crio-2d358b6ac53908eb50321375e34c9302fec1669c1a2f6a078f1343d83b6dbec2 WatchSource:0}: Error finding container 2d358b6ac53908eb50321375e34c9302fec1669c1a2f6a078f1343d83b6dbec2: Status 404 returned error can't find the container with id 2d358b6ac53908eb50321375e34c9302fec1669c1a2f6a078f1343d83b6dbec2 Sep 10 19:29:45.846859 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:45.846825 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql" event={"ID":"e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb","Type":"ContainerStarted","Data":"2d358b6ac53908eb50321375e34c9302fec1669c1a2f6a078f1343d83b6dbec2"} Sep 10 19:29:46.852702 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:46.852610 2578 generic.go:356] "Generic (PLEG): container finished" podID="e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb" containerID="6fc38629665d8c159b54e391729cfffcb7a08ea660b70ae77a92f40d3266de7a" exitCode=0 Sep 10 19:29:46.852702 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:46.852677 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql" event={"ID":"e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb","Type":"ContainerDied","Data":"6fc38629665d8c159b54e391729cfffcb7a08ea660b70ae77a92f40d3266de7a"} Sep 10 19:29:48.000886 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:48.000864 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql" Sep 10 19:29:48.097737 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:48.097704 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb-maas-api-service-ca\") pod \"e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb\" (UID: \"e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb\") " Sep 10 19:29:48.098048 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:48.098025 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb" (UID: "e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:29:48.199003 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:48.198969 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb-maas-api-service-ca\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:29:48.865422 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:48.865392 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql" Sep 10 19:29:48.865571 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:48.865426 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql" event={"ID":"e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb","Type":"ContainerDied","Data":"2d358b6ac53908eb50321375e34c9302fec1669c1a2f6a078f1343d83b6dbec2"} Sep 10 19:29:48.865571 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:48.865499 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d358b6ac53908eb50321375e34c9302fec1669c1a2f6a078f1343d83b6dbec2" Sep 10 19:29:49.417475 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:49.417446 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8"] Sep 10 19:29:49.417865 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:49.417811 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb" containerName="revoke-keys" Sep 10 19:29:49.417865 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:49.417823 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb" containerName="revoke-keys" Sep 10 19:29:49.417935 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:49.417891 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb" containerName="revoke-keys" Sep 10 19:29:49.420866 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:49.420841 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8" Sep 10 19:29:49.423115 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:49.423094 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkgt8\"" Sep 10 19:29:49.423237 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:49.423095 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:29:49.430336 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:49.430311 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8"] Sep 10 19:29:49.509790 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:49.509752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e739b656-d72d-43d1-8bcf-ff074d17b864-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b3108b-cz7b8\" (UID: \"e739b656-d72d-43d1-8bcf-ff074d17b864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8" Sep 10 19:29:49.610696 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:49.610660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e739b656-d72d-43d1-8bcf-ff074d17b864-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b3108b-cz7b8\" (UID: \"e739b656-d72d-43d1-8bcf-ff074d17b864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8" Sep 10 19:29:49.611354 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:49.611334 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e739b656-d72d-43d1-8bcf-ff074d17b864-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b3108b-cz7b8\" (UID: \"e739b656-d72d-43d1-8bcf-ff074d17b864\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8" Sep 10 19:29:49.732773 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:49.732657 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8" Sep 10 19:29:50.003974 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:50.003903 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8"] Sep 10 19:29:50.008198 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:29:50.008172 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode739b656_d72d_43d1_8bcf_ff074d17b864.slice/crio-d1b7f62ef9fcbfd2a4a47a99e68e2fdf99a94dcb35fc272b3b3a49984191a2a3 WatchSource:0}: Error finding container d1b7f62ef9fcbfd2a4a47a99e68e2fdf99a94dcb35fc272b3b3a49984191a2a3: Status 404 returned error can't find the container with id d1b7f62ef9fcbfd2a4a47a99e68e2fdf99a94dcb35fc272b3b3a49984191a2a3 Sep 10 19:29:50.876070 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:50.876039 2578 generic.go:356] "Generic (PLEG): container finished" podID="e739b656-d72d-43d1-8bcf-ff074d17b864" containerID="cd394e5b1a549ddd136ea38322f4298d7ea2ec4705b4d9f6e786cb6f4b00b550" exitCode=0 Sep 10 19:29:50.876426 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:50.876079 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8" event={"ID":"e739b656-d72d-43d1-8bcf-ff074d17b864","Type":"ContainerDied","Data":"cd394e5b1a549ddd136ea38322f4298d7ea2ec4705b4d9f6e786cb6f4b00b550"} Sep 10 19:29:50.876426 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:50.876117 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8" event={"ID":"e739b656-d72d-43d1-8bcf-ff074d17b864","Type":"ContainerStarted","Data":"d1b7f62ef9fcbfd2a4a47a99e68e2fdf99a94dcb35fc272b3b3a49984191a2a3"} Sep 10 19:29:51.033502 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.033419 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn"] Sep 10 19:29:51.033684 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.033657 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" podUID="cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a" containerName="payload-processing" containerID="cri-o://87811b8221e35949204c980009ad1f8a5381bbe8ab68583a881ba12eeb45ec0d" gracePeriod=30 Sep 10 19:29:51.045879 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.045851 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww"] Sep 10 19:29:51.046559 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.046078 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" podUID="9cb75273-a0fd-40fb-b8c1-33ac13c7c661" containerName="payload-pre-processing" containerID="cri-o://181b4685c8660f92f986777ab8335d3af5e5370cd479b6c6e0ec9bac78d67dd0" gracePeriod=30 Sep 10 19:29:51.223592 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.223566 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql"] Sep 10 19:29:51.228018 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.227991 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0b1e2fe-mmlql"] Sep 10 19:29:51.296911 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.296842 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" Sep 10 19:29:51.325828 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.325807 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" Sep 10 19:29:51.395107 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.395076 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb" path="/var/lib/kubelet/pods/e6cc24b4-a345-4501-b8a7-6e5a6c80ddcb/volumes" Sep 10 19:29:51.428454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.428429 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb75273-a0fd-40fb-b8c1-33ac13c7c661-plugins-config-volume\") pod \"9cb75273-a0fd-40fb-b8c1-33ac13c7c661\" (UID: \"9cb75273-a0fd-40fb-b8c1-33ac13c7c661\") " Sep 10 19:29:51.428454 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.428459 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr7rj\" (UniqueName: \"kubernetes.io/projected/9cb75273-a0fd-40fb-b8c1-33ac13c7c661-kube-api-access-nr7rj\") pod \"9cb75273-a0fd-40fb-b8c1-33ac13c7c661\" (UID: \"9cb75273-a0fd-40fb-b8c1-33ac13c7c661\") " Sep 10 19:29:51.428654 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.428480 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a-plugins-config-volume\") pod \"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a\" (UID: \"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a\") " Sep 10 19:29:51.428654 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.428506 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brwqv\" (UniqueName: \"kubernetes.io/projected/cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a-kube-api-access-brwqv\") pod \"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a\" (UID: \"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a\") " Sep 10 19:29:51.428903 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.428880 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb75273-a0fd-40fb-b8c1-33ac13c7c661-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9cb75273-a0fd-40fb-b8c1-33ac13c7c661" (UID: "9cb75273-a0fd-40fb-b8c1-33ac13c7c661"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:29:51.428903 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.428889 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a" (UID: "cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:29:51.430622 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.430600 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a-kube-api-access-brwqv" (OuterVolumeSpecName: "kube-api-access-brwqv") pod "cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a" (UID: "cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a"). InnerVolumeSpecName "kube-api-access-brwqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:29:51.430698 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.430620 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb75273-a0fd-40fb-b8c1-33ac13c7c661-kube-api-access-nr7rj" (OuterVolumeSpecName: "kube-api-access-nr7rj") pod "9cb75273-a0fd-40fb-b8c1-33ac13c7c661" (UID: "9cb75273-a0fd-40fb-b8c1-33ac13c7c661"). InnerVolumeSpecName "kube-api-access-nr7rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:29:51.529676 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.529646 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cb75273-a0fd-40fb-b8c1-33ac13c7c661-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:29:51.529676 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.529673 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr7rj\" (UniqueName: \"kubernetes.io/projected/9cb75273-a0fd-40fb-b8c1-33ac13c7c661-kube-api-access-nr7rj\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:29:51.529876 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.529686 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:29:51.529876 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.529699 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brwqv\" (UniqueName: \"kubernetes.io/projected/cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a-kube-api-access-brwqv\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:29:51.882712 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.882675 2578 generic.go:356] "Generic (PLEG): container finished" podID="9cb75273-a0fd-40fb-b8c1-33ac13c7c661" containerID="181b4685c8660f92f986777ab8335d3af5e5370cd479b6c6e0ec9bac78d67dd0" exitCode=0 Sep 10 19:29:51.883160 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.882765 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" Sep 10 19:29:51.883160 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.882773 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" event={"ID":"9cb75273-a0fd-40fb-b8c1-33ac13c7c661","Type":"ContainerDied","Data":"181b4685c8660f92f986777ab8335d3af5e5370cd479b6c6e0ec9bac78d67dd0"} Sep 10 19:29:51.883160 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.882859 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww" event={"ID":"9cb75273-a0fd-40fb-b8c1-33ac13c7c661","Type":"ContainerDied","Data":"98f710f24bb5ce758ec6cb85c94fcf64f2ff24f2f4a781dcfdc2febecc2c227d"} Sep 10 19:29:51.883160 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.882890 2578 scope.go:117] "RemoveContainer" containerID="181b4685c8660f92f986777ab8335d3af5e5370cd479b6c6e0ec9bac78d67dd0" Sep 10 19:29:51.884402 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.884379 2578 generic.go:356] "Generic (PLEG): container finished" podID="cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a" containerID="87811b8221e35949204c980009ad1f8a5381bbe8ab68583a881ba12eeb45ec0d" exitCode=0 Sep 10 19:29:51.884530 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.884427 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" Sep 10 19:29:51.884530 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.884462 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" event={"ID":"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a","Type":"ContainerDied","Data":"87811b8221e35949204c980009ad1f8a5381bbe8ab68583a881ba12eeb45ec0d"} Sep 10 19:29:51.884530 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.884516 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn" event={"ID":"cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a","Type":"ContainerDied","Data":"a136e0db0862f39039130407e23cdc7efded1d90ee62fb8eba3d2fd254598f95"} Sep 10 19:29:51.895838 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.895819 2578 scope.go:117] "RemoveContainer" containerID="181b4685c8660f92f986777ab8335d3af5e5370cd479b6c6e0ec9bac78d67dd0" Sep 10 19:29:51.896237 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:29:51.896213 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"181b4685c8660f92f986777ab8335d3af5e5370cd479b6c6e0ec9bac78d67dd0\": container with ID starting with 181b4685c8660f92f986777ab8335d3af5e5370cd479b6c6e0ec9bac78d67dd0 not found: ID does not exist" containerID="181b4685c8660f92f986777ab8335d3af5e5370cd479b6c6e0ec9bac78d67dd0" Sep 10 19:29:51.896337 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.896246 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"181b4685c8660f92f986777ab8335d3af5e5370cd479b6c6e0ec9bac78d67dd0"} err="failed to get container status \"181b4685c8660f92f986777ab8335d3af5e5370cd479b6c6e0ec9bac78d67dd0\": rpc error: code = NotFound desc = could not find container \"181b4685c8660f92f986777ab8335d3af5e5370cd479b6c6e0ec9bac78d67dd0\": container with ID starting with 181b4685c8660f92f986777ab8335d3af5e5370cd479b6c6e0ec9bac78d67dd0 not found: ID does not exist" Sep 10 19:29:51.896337 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.896271 2578 scope.go:117] "RemoveContainer" containerID="87811b8221e35949204c980009ad1f8a5381bbe8ab68583a881ba12eeb45ec0d" Sep 10 19:29:51.904939 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.904916 2578 scope.go:117] "RemoveContainer" containerID="87811b8221e35949204c980009ad1f8a5381bbe8ab68583a881ba12eeb45ec0d" Sep 10 19:29:51.905278 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:29:51.905248 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87811b8221e35949204c980009ad1f8a5381bbe8ab68583a881ba12eeb45ec0d\": container with ID starting with 87811b8221e35949204c980009ad1f8a5381bbe8ab68583a881ba12eeb45ec0d not found: ID does not exist" containerID="87811b8221e35949204c980009ad1f8a5381bbe8ab68583a881ba12eeb45ec0d" Sep 10 19:29:51.905346 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.905287 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87811b8221e35949204c980009ad1f8a5381bbe8ab68583a881ba12eeb45ec0d"} err="failed to get container status \"87811b8221e35949204c980009ad1f8a5381bbe8ab68583a881ba12eeb45ec0d\": rpc error: code = NotFound desc = could not find container \"87811b8221e35949204c980009ad1f8a5381bbe8ab68583a881ba12eeb45ec0d\": container with ID starting with 87811b8221e35949204c980009ad1f8a5381bbe8ab68583a881ba12eeb45ec0d not found: ID does not exist" Sep 10 19:29:51.909097 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.909077 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww"] Sep 10 19:29:51.913197 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.913174 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f0b1e2fe-6f79c85786-62pww"] Sep 10 19:29:51.922268 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.922248 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn"] Sep 10 19:29:51.928710 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:51.928687 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f0b1e2fe-5cdb7488d5-bvgkn"] Sep 10 19:29:52.011364 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:52.011338 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8" Sep 10 19:29:52.133927 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:52.133859 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e739b656-d72d-43d1-8bcf-ff074d17b864-maas-api-service-ca\") pod \"e739b656-d72d-43d1-8bcf-ff074d17b864\" (UID: \"e739b656-d72d-43d1-8bcf-ff074d17b864\") " Sep 10 19:29:52.134242 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:52.134220 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e739b656-d72d-43d1-8bcf-ff074d17b864-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e739b656-d72d-43d1-8bcf-ff074d17b864" (UID: "e739b656-d72d-43d1-8bcf-ff074d17b864"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:29:52.234880 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:52.234850 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e739b656-d72d-43d1-8bcf-ff074d17b864-maas-api-service-ca\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:29:52.892241 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:52.892206 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8" Sep 10 19:29:52.892686 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:52.892213 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8" event={"ID":"e739b656-d72d-43d1-8bcf-ff074d17b864","Type":"ContainerDied","Data":"d1b7f62ef9fcbfd2a4a47a99e68e2fdf99a94dcb35fc272b3b3a49984191a2a3"} Sep 10 19:29:52.892686 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:52.892350 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1b7f62ef9fcbfd2a4a47a99e68e2fdf99a94dcb35fc272b3b3a49984191a2a3" Sep 10 19:29:53.391142 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:53.391107 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cb75273-a0fd-40fb-b8c1-33ac13c7c661" path="/var/lib/kubelet/pods/9cb75273-a0fd-40fb-b8c1-33ac13c7c661/volumes" Sep 10 19:29:53.391598 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:53.391574 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a" path="/var/lib/kubelet/pods/cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a/volumes" Sep 10 19:29:57.391167 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.391133 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d"] Sep 10 19:29:57.391602 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.391442 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e739b656-d72d-43d1-8bcf-ff074d17b864" containerName="revoke-keys" Sep 10 19:29:57.391602 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.391452 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e739b656-d72d-43d1-8bcf-ff074d17b864" containerName="revoke-keys" Sep 10 19:29:57.391602 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.391482 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cb75273-a0fd-40fb-b8c1-33ac13c7c661" containerName="payload-pre-processing" Sep 10 19:29:57.391602 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.391488 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cb75273-a0fd-40fb-b8c1-33ac13c7c661" containerName="payload-pre-processing" Sep 10 19:29:57.391602 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.391494 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a" containerName="payload-processing" Sep 10 19:29:57.391602 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.391498 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a" containerName="payload-processing" Sep 10 19:29:57.391602 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.391550 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb2e1cf5-03ba-4a8a-b7e2-56819ba30a0a" containerName="payload-processing" Sep 10 19:29:57.391602 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.391557 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e739b656-d72d-43d1-8bcf-ff074d17b864" containerName="revoke-keys" Sep 10 19:29:57.391602 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.391563 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cb75273-a0fd-40fb-b8c1-33ac13c7c661" containerName="payload-pre-processing" Sep 10 19:29:57.395872 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.395850 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" Sep 10 19:29:57.398016 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.397987 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-588a50a1\"" Sep 10 19:29:57.402742 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.402701 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d"] Sep 10 19:29:57.461776 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.461736 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz"] Sep 10 19:29:57.465095 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.465074 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" Sep 10 19:29:57.467831 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.467815 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-588a50a1-dockercfg-rd8wn\"" Sep 10 19:29:57.474654 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.474635 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz"] Sep 10 19:29:57.481627 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.481605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a39a1a-130d-4ee9-a7a3-9c8822db3789-plugins-config-volume\") pod \"payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d\" (UID: \"01a39a1a-130d-4ee9-a7a3-9c8822db3789\") " pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" Sep 10 19:29:57.481713 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.481668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrztg\" (UniqueName: \"kubernetes.io/projected/01a39a1a-130d-4ee9-a7a3-9c8822db3789-kube-api-access-wrztg\") pod \"payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d\" (UID: \"01a39a1a-130d-4ee9-a7a3-9c8822db3789\") " pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" Sep 10 19:29:57.582807 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.582770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92f692d-999c-42dc-ba6d-2225ffd0dceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz\" (UID: \"f92f692d-999c-42dc-ba6d-2225ffd0dceb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" Sep 10 19:29:57.582989 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.582824 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a39a1a-130d-4ee9-a7a3-9c8822db3789-plugins-config-volume\") pod \"payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d\" (UID: \"01a39a1a-130d-4ee9-a7a3-9c8822db3789\") " pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" Sep 10 19:29:57.582989 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.582932 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx7kv\" (UniqueName: \"kubernetes.io/projected/f92f692d-999c-42dc-ba6d-2225ffd0dceb-kube-api-access-fx7kv\") pod \"payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz\" (UID: \"f92f692d-999c-42dc-ba6d-2225ffd0dceb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" Sep 10 19:29:57.583101 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.582986 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrztg\" (UniqueName: \"kubernetes.io/projected/01a39a1a-130d-4ee9-a7a3-9c8822db3789-kube-api-access-wrztg\") pod \"payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d\" (UID: \"01a39a1a-130d-4ee9-a7a3-9c8822db3789\") " pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" Sep 10 19:29:57.583491 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.583472 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a39a1a-130d-4ee9-a7a3-9c8822db3789-plugins-config-volume\") pod \"payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d\" (UID: \"01a39a1a-130d-4ee9-a7a3-9c8822db3789\") " pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" Sep 10 19:29:57.590893 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.590867 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrztg\" (UniqueName: \"kubernetes.io/projected/01a39a1a-130d-4ee9-a7a3-9c8822db3789-kube-api-access-wrztg\") pod \"payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d\" (UID: \"01a39a1a-130d-4ee9-a7a3-9c8822db3789\") " pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" Sep 10 19:29:57.684370 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.684290 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92f692d-999c-42dc-ba6d-2225ffd0dceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz\" (UID: \"f92f692d-999c-42dc-ba6d-2225ffd0dceb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" Sep 10 19:29:57.684513 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.684389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7kv\" (UniqueName: \"kubernetes.io/projected/f92f692d-999c-42dc-ba6d-2225ffd0dceb-kube-api-access-fx7kv\") pod \"payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz\" (UID: \"f92f692d-999c-42dc-ba6d-2225ffd0dceb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" Sep 10 19:29:57.685037 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.685018 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92f692d-999c-42dc-ba6d-2225ffd0dceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz\" (UID: \"f92f692d-999c-42dc-ba6d-2225ffd0dceb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" Sep 10 19:29:57.692629 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.692594 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7kv\" (UniqueName: \"kubernetes.io/projected/f92f692d-999c-42dc-ba6d-2225ffd0dceb-kube-api-access-fx7kv\") pod \"payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz\" (UID: \"f92f692d-999c-42dc-ba6d-2225ffd0dceb\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" Sep 10 19:29:57.707441 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.707421 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" Sep 10 19:29:57.776528 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.776489 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" Sep 10 19:29:57.983558 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:57.983534 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d"] Sep 10 19:29:57.985535 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:29:57.985507 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a39a1a_130d_4ee9_a7a3_9c8822db3789.slice/crio-e957d43ca151a49c017bcd061f2b2f31a2471fc0418a625ccdb04ca9ea11af82 WatchSource:0}: Error finding container e957d43ca151a49c017bcd061f2b2f31a2471fc0418a625ccdb04ca9ea11af82: Status 404 returned error can't find the container with id e957d43ca151a49c017bcd061f2b2f31a2471fc0418a625ccdb04ca9ea11af82 Sep 10 19:29:58.050328 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:58.050305 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz"] Sep 10 19:29:58.052993 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:29:58.052967 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92f692d_999c_42dc_ba6d_2225ffd0dceb.slice/crio-d3e2f086c885cba921b578e67783569fc536e2ff06ba5fe81642d5c2224f96cf WatchSource:0}: Error finding container d3e2f086c885cba921b578e67783569fc536e2ff06ba5fe81642d5c2224f96cf: Status 404 returned error can't find the container with id d3e2f086c885cba921b578e67783569fc536e2ff06ba5fe81642d5c2224f96cf Sep 10 19:29:58.927278 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:58.927241 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" event={"ID":"f92f692d-999c-42dc-ba6d-2225ffd0dceb","Type":"ContainerStarted","Data":"b94515f8acc4b133e9c6505e9728a501e36d280888f639e37d1405cbc60ab5d9"} Sep 10 19:29:58.927794 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:58.927303 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" event={"ID":"f92f692d-999c-42dc-ba6d-2225ffd0dceb","Type":"ContainerStarted","Data":"d3e2f086c885cba921b578e67783569fc536e2ff06ba5fe81642d5c2224f96cf"} Sep 10 19:29:58.927794 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:58.927348 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" Sep 10 19:29:58.929117 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:58.929094 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" event={"ID":"01a39a1a-130d-4ee9-a7a3-9c8822db3789","Type":"ContainerStarted","Data":"07ba4cd4cb22a47b6d2e33529b3f9b93406c95c75bbc823cfd15d19c20a1f0dd"} Sep 10 19:29:58.929231 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:58.929137 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" event={"ID":"01a39a1a-130d-4ee9-a7a3-9c8822db3789","Type":"ContainerStarted","Data":"e957d43ca151a49c017bcd061f2b2f31a2471fc0418a625ccdb04ca9ea11af82"} Sep 10 19:29:58.929231 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:58.929159 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" Sep 10 19:29:58.943278 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:58.943234 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" podStartSLOduration=1.943218827 podStartE2EDuration="1.943218827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:29:58.942613961 +0000 UTC m=+848.073115593" watchObservedRunningTime="2026-09-10 19:29:58.943218827 +0000 UTC m=+848.073720462" Sep 10 19:29:58.960518 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:58.960450 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" podStartSLOduration=1.960432978 podStartE2EDuration="1.960432978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:29:58.95932407 +0000 UTC m=+848.089825700" watchObservedRunningTime="2026-09-10 19:29:58.960432978 +0000 UTC m=+848.090934610" Sep 10 19:29:59.854385 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:59.854352 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg"] Sep 10 19:29:59.854602 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:59.854578 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" podUID="4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10" containerName="payload-processing" containerID="cri-o://8f9c486537dc3cac9f94d8636836d6369b2f7a80c6b08880e9a8ccae2227bdfb" gracePeriod=30 Sep 10 19:29:59.863238 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:59.863210 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr"] Sep 10 19:29:59.863468 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:29:59.863436 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" podUID="5c249300-612f-4716-ba16-9d3a7c48c08a" containerName="payload-pre-processing" containerID="cri-o://ffaf65659fdb78966bcea6d00b3d2ff514a13c6307763019dfb167f36fa55143" gracePeriod=30 Sep 10 19:30:00.083089 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.083060 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8"] Sep 10 19:30:00.085778 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.085748 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b3108b-cz7b8"] Sep 10 19:30:00.147220 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.147192 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" Sep 10 19:30:00.149594 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.149570 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817810-88dgw"] Sep 10 19:30:00.149926 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.149911 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c249300-612f-4716-ba16-9d3a7c48c08a" containerName="payload-pre-processing" Sep 10 19:30:00.149926 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.149927 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c249300-612f-4716-ba16-9d3a7c48c08a" containerName="payload-pre-processing" Sep 10 19:30:00.150055 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.149987 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c249300-612f-4716-ba16-9d3a7c48c08a" containerName="payload-pre-processing" Sep 10 19:30:00.150311 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.150295 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" Sep 10 19:30:00.153032 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.153005 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-88dgw" Sep 10 19:30:00.154589 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.154571 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc"] Sep 10 19:30:00.154913 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.154900 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10" containerName="payload-processing" Sep 10 19:30:00.154976 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.154914 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10" containerName="payload-processing" Sep 10 19:30:00.154976 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.154962 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10" containerName="payload-processing" Sep 10 19:30:00.155309 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.155291 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkgt8\"" Sep 10 19:30:00.157976 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.157959 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc" Sep 10 19:30:00.162853 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.162699 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-88dgw" Sep 10 19:30:00.165930 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.165908 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817810-88dgw"] Sep 10 19:30:00.173648 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.173621 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc" Sep 10 19:30:00.183745 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.183695 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc"] Sep 10 19:30:00.308304 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.308272 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c249300-612f-4716-ba16-9d3a7c48c08a-plugins-config-volume\") pod \"5c249300-612f-4716-ba16-9d3a7c48c08a\" (UID: \"5c249300-612f-4716-ba16-9d3a7c48c08a\") " Sep 10 19:30:00.308446 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.308344 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10-plugins-config-volume\") pod \"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10\" (UID: \"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10\") " Sep 10 19:30:00.308446 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.308374 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txm56\" (UniqueName: \"kubernetes.io/projected/5c249300-612f-4716-ba16-9d3a7c48c08a-kube-api-access-txm56\") pod \"5c249300-612f-4716-ba16-9d3a7c48c08a\" (UID: \"5c249300-612f-4716-ba16-9d3a7c48c08a\") " Sep 10 19:30:00.308549 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.308456 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mkdm\" (UniqueName: \"kubernetes.io/projected/4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10-kube-api-access-6mkdm\") pod \"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10\" (UID: \"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10\") " Sep 10 19:30:00.308958 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.308754 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c249300-612f-4716-ba16-9d3a7c48c08a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c249300-612f-4716-ba16-9d3a7c48c08a" (UID: "5c249300-612f-4716-ba16-9d3a7c48c08a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:30:00.308958 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.308765 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10" (UID: "4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:30:00.310865 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.310835 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c249300-612f-4716-ba16-9d3a7c48c08a-kube-api-access-txm56" (OuterVolumeSpecName: "kube-api-access-txm56") pod "5c249300-612f-4716-ba16-9d3a7c48c08a" (UID: "5c249300-612f-4716-ba16-9d3a7c48c08a"). InnerVolumeSpecName "kube-api-access-txm56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:30:00.311246 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.311221 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10-kube-api-access-6mkdm" (OuterVolumeSpecName: "kube-api-access-6mkdm") pod "4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10" (UID: "4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10"). InnerVolumeSpecName "kube-api-access-6mkdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:30:00.409527 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.409474 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c249300-612f-4716-ba16-9d3a7c48c08a-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:30:00.409527 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.409497 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:30:00.409527 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.409506 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txm56\" (UniqueName: \"kubernetes.io/projected/5c249300-612f-4716-ba16-9d3a7c48c08a-kube-api-access-txm56\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:30:00.409527 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.409515 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mkdm\" (UniqueName: \"kubernetes.io/projected/4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10-kube-api-access-6mkdm\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:30:00.456094 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.455996 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817810-88dgw"] Sep 10 19:30:00.458711 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:30:00.458685 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe9602d_36f1_48a2_bfaf_cea8b2b039b7.slice/crio-172c4cca6a4cd1b86d170cdcc78d57550a722c9a0732513b89db8bbd3111782c WatchSource:0}: Error finding container 172c4cca6a4cd1b86d170cdcc78d57550a722c9a0732513b89db8bbd3111782c: Status 404 returned error can't find the container with id 172c4cca6a4cd1b86d170cdcc78d57550a722c9a0732513b89db8bbd3111782c Sep 10 19:30:00.466020 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.465967 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc"] Sep 10 19:30:00.467782 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:30:00.467761 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a7dbb0_aff0_4c57_8650_41d819bea794.slice/crio-3386635d0846c3befa9f9b34bd7bd83823811d18e4d6dd3d4a690b5ed1078d35 WatchSource:0}: Error finding container 3386635d0846c3befa9f9b34bd7bd83823811d18e4d6dd3d4a690b5ed1078d35: Status 404 returned error can't find the container with id 3386635d0846c3befa9f9b34bd7bd83823811d18e4d6dd3d4a690b5ed1078d35 Sep 10 19:30:00.942491 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.942454 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc" event={"ID":"a7a7dbb0-aff0-4c57-8650-41d819bea794","Type":"ContainerStarted","Data":"f7498ca014b8851560092b3defb1efe6edda467b43237261795cb5ed5057d859"} Sep 10 19:30:00.942649 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.942510 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc" event={"ID":"a7a7dbb0-aff0-4c57-8650-41d819bea794","Type":"ContainerStarted","Data":"3386635d0846c3befa9f9b34bd7bd83823811d18e4d6dd3d4a690b5ed1078d35"} Sep 10 19:30:00.943717 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.943695 2578 generic.go:356] "Generic (PLEG): container finished" podID="4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10" containerID="8f9c486537dc3cac9f94d8636836d6369b2f7a80c6b08880e9a8ccae2227bdfb" exitCode=0 Sep 10 19:30:00.943846 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.943761 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" Sep 10 19:30:00.943846 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.943766 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" event={"ID":"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10","Type":"ContainerDied","Data":"8f9c486537dc3cac9f94d8636836d6369b2f7a80c6b08880e9a8ccae2227bdfb"} Sep 10 19:30:00.943846 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.943818 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg" event={"ID":"4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10","Type":"ContainerDied","Data":"9908112db1863bf7e86f98717fad75d9ddfca5c223acd1b4c21da4ad91906178"} Sep 10 19:30:00.944004 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.943851 2578 scope.go:117] "RemoveContainer" containerID="8f9c486537dc3cac9f94d8636836d6369b2f7a80c6b08880e9a8ccae2227bdfb" Sep 10 19:30:00.945173 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.945144 2578 generic.go:356] "Generic (PLEG): container finished" podID="5c249300-612f-4716-ba16-9d3a7c48c08a" containerID="ffaf65659fdb78966bcea6d00b3d2ff514a13c6307763019dfb167f36fa55143" exitCode=0 Sep 10 19:30:00.945173 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.945179 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" event={"ID":"5c249300-612f-4716-ba16-9d3a7c48c08a","Type":"ContainerDied","Data":"ffaf65659fdb78966bcea6d00b3d2ff514a13c6307763019dfb167f36fa55143"} Sep 10 19:30:00.945382 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.945235 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" Sep 10 19:30:00.945382 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.945236 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr" event={"ID":"5c249300-612f-4716-ba16-9d3a7c48c08a","Type":"ContainerDied","Data":"4f4429bdc338ae2185283fc8f1bcd996654bec004884f4acf9c79895a7f98fb9"} Sep 10 19:30:00.947120 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.947091 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-88dgw" event={"ID":"4fe9602d-36f1-48a2-bfaf-cea8b2b039b7","Type":"ContainerStarted","Data":"95180f7102e9900f1f57f96774f5c00618177823a9ae85c55a8f64939c150a99"} Sep 10 19:30:00.947214 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.947141 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-88dgw" event={"ID":"4fe9602d-36f1-48a2-bfaf-cea8b2b039b7","Type":"ContainerStarted","Data":"172c4cca6a4cd1b86d170cdcc78d57550a722c9a0732513b89db8bbd3111782c"} Sep 10 19:30:00.952619 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.952600 2578 scope.go:117] "RemoveContainer" containerID="8f9c486537dc3cac9f94d8636836d6369b2f7a80c6b08880e9a8ccae2227bdfb" Sep 10 19:30:00.952932 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:30:00.952913 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9c486537dc3cac9f94d8636836d6369b2f7a80c6b08880e9a8ccae2227bdfb\": container with ID starting with 8f9c486537dc3cac9f94d8636836d6369b2f7a80c6b08880e9a8ccae2227bdfb not found: ID does not exist" containerID="8f9c486537dc3cac9f94d8636836d6369b2f7a80c6b08880e9a8ccae2227bdfb" Sep 10 19:30:00.953004 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.952944 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9c486537dc3cac9f94d8636836d6369b2f7a80c6b08880e9a8ccae2227bdfb"} err="failed to get container status \"8f9c486537dc3cac9f94d8636836d6369b2f7a80c6b08880e9a8ccae2227bdfb\": rpc error: code = NotFound desc = could not find container \"8f9c486537dc3cac9f94d8636836d6369b2f7a80c6b08880e9a8ccae2227bdfb\": container with ID starting with 8f9c486537dc3cac9f94d8636836d6369b2f7a80c6b08880e9a8ccae2227bdfb not found: ID does not exist" Sep 10 19:30:00.953004 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.952968 2578 scope.go:117] "RemoveContainer" containerID="ffaf65659fdb78966bcea6d00b3d2ff514a13c6307763019dfb167f36fa55143" Sep 10 19:30:00.961932 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.961621 2578 scope.go:117] "RemoveContainer" containerID="ffaf65659fdb78966bcea6d00b3d2ff514a13c6307763019dfb167f36fa55143" Sep 10 19:30:00.962298 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:30:00.962263 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffaf65659fdb78966bcea6d00b3d2ff514a13c6307763019dfb167f36fa55143\": container with ID starting with ffaf65659fdb78966bcea6d00b3d2ff514a13c6307763019dfb167f36fa55143 not found: ID does not exist" containerID="ffaf65659fdb78966bcea6d00b3d2ff514a13c6307763019dfb167f36fa55143" Sep 10 19:30:00.962364 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.962308 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffaf65659fdb78966bcea6d00b3d2ff514a13c6307763019dfb167f36fa55143"} err="failed to get container status \"ffaf65659fdb78966bcea6d00b3d2ff514a13c6307763019dfb167f36fa55143\": rpc error: code = NotFound desc = could not find container \"ffaf65659fdb78966bcea6d00b3d2ff514a13c6307763019dfb167f36fa55143\": container with ID starting with ffaf65659fdb78966bcea6d00b3d2ff514a13c6307763019dfb167f36fa55143 not found: ID does not exist" Sep 10 19:30:00.963490 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.963455 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc" podStartSLOduration=0.963443486 podStartE2EDuration="963.443486ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:30:00.961306953 +0000 UTC m=+850.091808586" watchObservedRunningTime="2026-09-10 19:30:00.963443486 +0000 UTC m=+850.093945127" Sep 10 19:30:00.976365 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.976346 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg"] Sep 10 19:30:00.980321 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.980301 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b3108b-9449f7f77-z27gg"] Sep 10 19:30:00.992652 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:00.992602 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-88dgw" podStartSLOduration=0.992589412 podStartE2EDuration="992.589412ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:30:00.991104961 +0000 UTC m=+850.121606603" watchObservedRunningTime="2026-09-10 19:30:00.992589412 +0000 UTC m=+850.123091044" Sep 10 19:30:01.005058 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:01.005038 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr"] Sep 10 19:30:01.009702 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:01.009683 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b3108b-5c898cf69d-bqxcr"] Sep 10 19:30:01.391567 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:01.391530 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10" path="/var/lib/kubelet/pods/4b9b72f5-dc34-4fad-81fb-8a0c2a5faf10/volumes" Sep 10 19:30:01.392078 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:01.392057 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c249300-612f-4716-ba16-9d3a7c48c08a" path="/var/lib/kubelet/pods/5c249300-612f-4716-ba16-9d3a7c48c08a/volumes" Sep 10 19:30:01.392411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:01.392394 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e739b656-d72d-43d1-8bcf-ff074d17b864" path="/var/lib/kubelet/pods/e739b656-d72d-43d1-8bcf-ff074d17b864/volumes" Sep 10 19:30:01.954792 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:01.954756 2578 generic.go:356] "Generic (PLEG): container finished" podID="4fe9602d-36f1-48a2-bfaf-cea8b2b039b7" containerID="95180f7102e9900f1f57f96774f5c00618177823a9ae85c55a8f64939c150a99" exitCode=0 Sep 10 19:30:01.954968 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:01.954826 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-88dgw" event={"ID":"4fe9602d-36f1-48a2-bfaf-cea8b2b039b7","Type":"ContainerDied","Data":"95180f7102e9900f1f57f96774f5c00618177823a9ae85c55a8f64939c150a99"} Sep 10 19:30:01.956311 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:01.956286 2578 generic.go:356] "Generic (PLEG): container finished" podID="a7a7dbb0-aff0-4c57-8650-41d819bea794" containerID="f7498ca014b8851560092b3defb1efe6edda467b43237261795cb5ed5057d859" exitCode=0 Sep 10 19:30:01.956420 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:01.956350 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc" event={"ID":"a7a7dbb0-aff0-4c57-8650-41d819bea794","Type":"ContainerDied","Data":"f7498ca014b8851560092b3defb1efe6edda467b43237261795cb5ed5057d859"} Sep 10 19:30:03.117389 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:03.117364 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-88dgw" Sep 10 19:30:03.138021 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:03.137998 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc" Sep 10 19:30:03.967947 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:03.967914 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-88dgw" event={"ID":"4fe9602d-36f1-48a2-bfaf-cea8b2b039b7","Type":"ContainerDied","Data":"172c4cca6a4cd1b86d170cdcc78d57550a722c9a0732513b89db8bbd3111782c"} Sep 10 19:30:03.968156 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:03.967971 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="172c4cca6a4cd1b86d170cdcc78d57550a722c9a0732513b89db8bbd3111782c" Sep 10 19:30:03.968156 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:03.967929 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817810-88dgw" Sep 10 19:30:03.969250 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:03.969229 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc" Sep 10 19:30:03.969379 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:03.969276 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc" event={"ID":"a7a7dbb0-aff0-4c57-8650-41d819bea794","Type":"ContainerDied","Data":"3386635d0846c3befa9f9b34bd7bd83823811d18e4d6dd3d4a690b5ed1078d35"} Sep 10 19:30:03.969379 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:03.969315 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3386635d0846c3befa9f9b34bd7bd83823811d18e4d6dd3d4a690b5ed1078d35" Sep 10 19:30:09.938239 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:09.938204 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" Sep 10 19:30:09.938673 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:09.938371 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" Sep 10 19:30:12.432480 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.432448 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44"] Sep 10 19:30:12.432849 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.432833 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fe9602d-36f1-48a2-bfaf-cea8b2b039b7" containerName="cleanup" Sep 10 19:30:12.432902 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.432853 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fe9602d-36f1-48a2-bfaf-cea8b2b039b7" containerName="cleanup" Sep 10 19:30:12.432902 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.432865 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7a7dbb0-aff0-4c57-8650-41d819bea794" containerName="cleanup" Sep 10 19:30:12.432902 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.432871 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7a7dbb0-aff0-4c57-8650-41d819bea794" containerName="cleanup" Sep 10 19:30:12.432992 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.432939 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7a7dbb0-aff0-4c57-8650-41d819bea794" containerName="cleanup" Sep 10 19:30:12.432992 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.432948 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fe9602d-36f1-48a2-bfaf-cea8b2b039b7" containerName="cleanup" Sep 10 19:30:12.435687 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.435669 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" Sep 10 19:30:12.437966 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.437944 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-faed1b-dockercfg-5zjqz\"" Sep 10 19:30:12.438132 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.438115 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-faed1b\"" Sep 10 19:30:12.444626 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.444606 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44"] Sep 10 19:30:12.515648 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.515615 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85tzp\" (UniqueName: \"kubernetes.io/projected/8a6d133c-2eb7-4005-aa50-932853269cbd-kube-api-access-85tzp\") pod \"payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44\" (UID: \"8a6d133c-2eb7-4005-aa50-932853269cbd\") " pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" Sep 10 19:30:12.515815 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.515678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6d133c-2eb7-4005-aa50-932853269cbd-plugins-config-volume\") pod \"payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44\" (UID: \"8a6d133c-2eb7-4005-aa50-932853269cbd\") " pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" Sep 10 19:30:12.616842 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.616811 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85tzp\" (UniqueName: \"kubernetes.io/projected/8a6d133c-2eb7-4005-aa50-932853269cbd-kube-api-access-85tzp\") pod \"payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44\" (UID: \"8a6d133c-2eb7-4005-aa50-932853269cbd\") " pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" Sep 10 19:30:12.617023 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.616862 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6d133c-2eb7-4005-aa50-932853269cbd-plugins-config-volume\") pod \"payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44\" (UID: \"8a6d133c-2eb7-4005-aa50-932853269cbd\") " pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" Sep 10 19:30:12.617568 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.617548 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6d133c-2eb7-4005-aa50-932853269cbd-plugins-config-volume\") pod \"payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44\" (UID: \"8a6d133c-2eb7-4005-aa50-932853269cbd\") " pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" Sep 10 19:30:12.625067 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.625036 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85tzp\" (UniqueName: \"kubernetes.io/projected/8a6d133c-2eb7-4005-aa50-932853269cbd-kube-api-access-85tzp\") pod \"payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44\" (UID: \"8a6d133c-2eb7-4005-aa50-932853269cbd\") " pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" Sep 10 19:30:12.746873 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:12.746792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" Sep 10 19:30:13.019567 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:13.019491 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44"] Sep 10 19:30:13.022302 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:30:13.022276 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6d133c_2eb7_4005_aa50_932853269cbd.slice/crio-9282800bf9915f71bca8136f60211c098178aaafa2071dd48fc92477343f1d22 WatchSource:0}: Error finding container 9282800bf9915f71bca8136f60211c098178aaafa2071dd48fc92477343f1d22: Status 404 returned error can't find the container with id 9282800bf9915f71bca8136f60211c098178aaafa2071dd48fc92477343f1d22 Sep 10 19:30:14.019706 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:14.019672 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" event={"ID":"8a6d133c-2eb7-4005-aa50-932853269cbd","Type":"ContainerStarted","Data":"987e427d363f1de6577d5a6ba2dc33007115cc76be72ba63383686ec8bd0beb5"} Sep 10 19:30:14.020077 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:14.019766 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" Sep 10 19:30:14.020077 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:14.019787 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" event={"ID":"8a6d133c-2eb7-4005-aa50-932853269cbd","Type":"ContainerStarted","Data":"9282800bf9915f71bca8136f60211c098178aaafa2071dd48fc92477343f1d22"} Sep 10 19:30:14.034677 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:14.034625 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" podStartSLOduration=2.034604885 podStartE2EDuration="2.034604885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:30:14.034533443 +0000 UTC m=+863.165035075" watchObservedRunningTime="2026-09-10 19:30:14.034604885 +0000 UTC m=+863.165106518" Sep 10 19:30:25.026795 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:25.026763 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" Sep 10 19:30:25.433531 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:25.433498 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk"] Sep 10 19:30:25.437079 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:25.437062 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk" Sep 10 19:30:25.439475 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:25.439450 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:30:25.439581 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:25.439488 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkgt8\"" Sep 10 19:30:25.445258 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:25.445122 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk"] Sep 10 19:30:25.524895 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:25.524866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555799aa-0cf2-43dc-987e-53adaf87fdd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk\" (UID: \"555799aa-0cf2-43dc-987e-53adaf87fdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk" Sep 10 19:30:25.625266 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:25.625240 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555799aa-0cf2-43dc-987e-53adaf87fdd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk\" (UID: \"555799aa-0cf2-43dc-987e-53adaf87fdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk" Sep 10 19:30:25.625917 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:25.625898 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555799aa-0cf2-43dc-987e-53adaf87fdd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk\" (UID: \"555799aa-0cf2-43dc-987e-53adaf87fdd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk" Sep 10 19:30:25.748926 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:25.748847 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk" Sep 10 19:30:26.022066 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:26.022042 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk"] Sep 10 19:30:26.024261 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:30:26.024232 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod555799aa_0cf2_43dc_987e_53adaf87fdd2.slice/crio-7027a4cc800871fb35200a46d94ac9d206f0132a7a8f437ac65efbb7223e924b WatchSource:0}: Error finding container 7027a4cc800871fb35200a46d94ac9d206f0132a7a8f437ac65efbb7223e924b: Status 404 returned error can't find the container with id 7027a4cc800871fb35200a46d94ac9d206f0132a7a8f437ac65efbb7223e924b Sep 10 19:30:26.078483 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:26.078460 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk" event={"ID":"555799aa-0cf2-43dc-987e-53adaf87fdd2","Type":"ContainerStarted","Data":"7027a4cc800871fb35200a46d94ac9d206f0132a7a8f437ac65efbb7223e924b"} Sep 10 19:30:27.094971 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:27.094869 2578 generic.go:356] "Generic (PLEG): container finished" podID="555799aa-0cf2-43dc-987e-53adaf87fdd2" containerID="86b4e009e4735736dcb98caa74cb128f4a849219500d88d7b9f93d25ee4989f7" exitCode=0 Sep 10 19:30:27.094971 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:27.094924 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk" event={"ID":"555799aa-0cf2-43dc-987e-53adaf87fdd2","Type":"ContainerDied","Data":"86b4e009e4735736dcb98caa74cb128f4a849219500d88d7b9f93d25ee4989f7"} Sep 10 19:30:28.232585 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:28.232563 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk" Sep 10 19:30:28.348025 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:28.347994 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555799aa-0cf2-43dc-987e-53adaf87fdd2-maas-api-service-ca\") pod \"555799aa-0cf2-43dc-987e-53adaf87fdd2\" (UID: \"555799aa-0cf2-43dc-987e-53adaf87fdd2\") " Sep 10 19:30:28.348326 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:28.348303 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/555799aa-0cf2-43dc-987e-53adaf87fdd2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "555799aa-0cf2-43dc-987e-53adaf87fdd2" (UID: "555799aa-0cf2-43dc-987e-53adaf87fdd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:30:28.449430 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:28.449402 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/555799aa-0cf2-43dc-987e-53adaf87fdd2-maas-api-service-ca\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:30:29.106540 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:29.106506 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk" event={"ID":"555799aa-0cf2-43dc-987e-53adaf87fdd2","Type":"ContainerDied","Data":"7027a4cc800871fb35200a46d94ac9d206f0132a7a8f437ac65efbb7223e924b"} Sep 10 19:30:29.106540 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:29.106530 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk" Sep 10 19:30:29.106784 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:29.106553 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7027a4cc800871fb35200a46d94ac9d206f0132a7a8f437ac65efbb7223e924b" Sep 10 19:30:37.689518 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:37.689484 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d"] Sep 10 19:30:37.690002 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:37.689759 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" podUID="01a39a1a-130d-4ee9-a7a3-9c8822db3789" containerName="payload-processing" containerID="cri-o://07ba4cd4cb22a47b6d2e33529b3f9b93406c95c75bbc823cfd15d19c20a1f0dd" gracePeriod=30 Sep 10 19:30:37.699286 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:37.699017 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz"] Sep 10 19:30:37.699286 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:37.699249 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" podUID="f92f692d-999c-42dc-ba6d-2225ffd0dceb" containerName="payload-pre-processing" containerID="cri-o://b94515f8acc4b133e9c6505e9728a501e36d280888f639e37d1405cbc60ab5d9" gracePeriod=30 Sep 10 19:30:37.901672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:37.900528 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk"] Sep 10 19:30:37.904720 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:37.904695 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-588a50a1-jcrjk"] Sep 10 19:30:37.951241 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:37.951185 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" Sep 10 19:30:37.975341 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:37.975319 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" Sep 10 19:30:38.032208 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.032183 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrztg\" (UniqueName: \"kubernetes.io/projected/01a39a1a-130d-4ee9-a7a3-9c8822db3789-kube-api-access-wrztg\") pod \"01a39a1a-130d-4ee9-a7a3-9c8822db3789\" (UID: \"01a39a1a-130d-4ee9-a7a3-9c8822db3789\") " Sep 10 19:30:38.032386 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.032213 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92f692d-999c-42dc-ba6d-2225ffd0dceb-plugins-config-volume\") pod \"f92f692d-999c-42dc-ba6d-2225ffd0dceb\" (UID: \"f92f692d-999c-42dc-ba6d-2225ffd0dceb\") " Sep 10 19:30:38.032386 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.032239 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx7kv\" (UniqueName: \"kubernetes.io/projected/f92f692d-999c-42dc-ba6d-2225ffd0dceb-kube-api-access-fx7kv\") pod \"f92f692d-999c-42dc-ba6d-2225ffd0dceb\" (UID: \"f92f692d-999c-42dc-ba6d-2225ffd0dceb\") " Sep 10 19:30:38.032386 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.032297 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a39a1a-130d-4ee9-a7a3-9c8822db3789-plugins-config-volume\") pod \"01a39a1a-130d-4ee9-a7a3-9c8822db3789\" (UID: \"01a39a1a-130d-4ee9-a7a3-9c8822db3789\") " Sep 10 19:30:38.032719 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.032669 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f92f692d-999c-42dc-ba6d-2225ffd0dceb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f92f692d-999c-42dc-ba6d-2225ffd0dceb" (UID: "f92f692d-999c-42dc-ba6d-2225ffd0dceb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:30:38.032938 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.032909 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a39a1a-130d-4ee9-a7a3-9c8822db3789-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01a39a1a-130d-4ee9-a7a3-9c8822db3789" (UID: "01a39a1a-130d-4ee9-a7a3-9c8822db3789"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:30:38.035974 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.035946 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92f692d-999c-42dc-ba6d-2225ffd0dceb-kube-api-access-fx7kv" (OuterVolumeSpecName: "kube-api-access-fx7kv") pod "f92f692d-999c-42dc-ba6d-2225ffd0dceb" (UID: "f92f692d-999c-42dc-ba6d-2225ffd0dceb"). InnerVolumeSpecName "kube-api-access-fx7kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:30:38.037691 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.037670 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a39a1a-130d-4ee9-a7a3-9c8822db3789-kube-api-access-wrztg" (OuterVolumeSpecName: "kube-api-access-wrztg") pod "01a39a1a-130d-4ee9-a7a3-9c8822db3789" (UID: "01a39a1a-130d-4ee9-a7a3-9c8822db3789"). InnerVolumeSpecName "kube-api-access-wrztg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:30:38.133621 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.133584 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrztg\" (UniqueName: \"kubernetes.io/projected/01a39a1a-130d-4ee9-a7a3-9c8822db3789-kube-api-access-wrztg\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:30:38.133621 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.133613 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92f692d-999c-42dc-ba6d-2225ffd0dceb-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:30:38.133621 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.133623 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx7kv\" (UniqueName: \"kubernetes.io/projected/f92f692d-999c-42dc-ba6d-2225ffd0dceb-kube-api-access-fx7kv\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:30:38.133862 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.133634 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a39a1a-130d-4ee9-a7a3-9c8822db3789-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:30:38.155014 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.154983 2578 generic.go:356] "Generic (PLEG): container finished" podID="f92f692d-999c-42dc-ba6d-2225ffd0dceb" containerID="b94515f8acc4b133e9c6505e9728a501e36d280888f639e37d1405cbc60ab5d9" exitCode=0 Sep 10 19:30:38.155153 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.155023 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" Sep 10 19:30:38.155153 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.155070 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" event={"ID":"f92f692d-999c-42dc-ba6d-2225ffd0dceb","Type":"ContainerDied","Data":"b94515f8acc4b133e9c6505e9728a501e36d280888f639e37d1405cbc60ab5d9"} Sep 10 19:30:38.155153 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.155127 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz" event={"ID":"f92f692d-999c-42dc-ba6d-2225ffd0dceb","Type":"ContainerDied","Data":"d3e2f086c885cba921b578e67783569fc536e2ff06ba5fe81642d5c2224f96cf"} Sep 10 19:30:38.155303 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.155157 2578 scope.go:117] "RemoveContainer" containerID="b94515f8acc4b133e9c6505e9728a501e36d280888f639e37d1405cbc60ab5d9" Sep 10 19:30:38.156427 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.156403 2578 generic.go:356] "Generic (PLEG): container finished" podID="01a39a1a-130d-4ee9-a7a3-9c8822db3789" containerID="07ba4cd4cb22a47b6d2e33529b3f9b93406c95c75bbc823cfd15d19c20a1f0dd" exitCode=0 Sep 10 19:30:38.156523 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.156450 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" Sep 10 19:30:38.156523 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.156509 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" event={"ID":"01a39a1a-130d-4ee9-a7a3-9c8822db3789","Type":"ContainerDied","Data":"07ba4cd4cb22a47b6d2e33529b3f9b93406c95c75bbc823cfd15d19c20a1f0dd"} Sep 10 19:30:38.156630 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.156548 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d" event={"ID":"01a39a1a-130d-4ee9-a7a3-9c8822db3789","Type":"ContainerDied","Data":"e957d43ca151a49c017bcd061f2b2f31a2471fc0418a625ccdb04ca9ea11af82"} Sep 10 19:30:38.165926 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.165901 2578 scope.go:117] "RemoveContainer" containerID="b94515f8acc4b133e9c6505e9728a501e36d280888f639e37d1405cbc60ab5d9" Sep 10 19:30:38.168938 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:30:38.168904 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94515f8acc4b133e9c6505e9728a501e36d280888f639e37d1405cbc60ab5d9\": container with ID starting with b94515f8acc4b133e9c6505e9728a501e36d280888f639e37d1405cbc60ab5d9 not found: ID does not exist" containerID="b94515f8acc4b133e9c6505e9728a501e36d280888f639e37d1405cbc60ab5d9" Sep 10 19:30:38.169037 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.168951 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94515f8acc4b133e9c6505e9728a501e36d280888f639e37d1405cbc60ab5d9"} err="failed to get container status \"b94515f8acc4b133e9c6505e9728a501e36d280888f639e37d1405cbc60ab5d9\": rpc error: code = NotFound desc = could not find container \"b94515f8acc4b133e9c6505e9728a501e36d280888f639e37d1405cbc60ab5d9\": container with ID starting with b94515f8acc4b133e9c6505e9728a501e36d280888f639e37d1405cbc60ab5d9 not found: ID does not exist" Sep 10 19:30:38.169037 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.168972 2578 scope.go:117] "RemoveContainer" containerID="07ba4cd4cb22a47b6d2e33529b3f9b93406c95c75bbc823cfd15d19c20a1f0dd" Sep 10 19:30:38.178036 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.177994 2578 scope.go:117] "RemoveContainer" containerID="07ba4cd4cb22a47b6d2e33529b3f9b93406c95c75bbc823cfd15d19c20a1f0dd" Sep 10 19:30:38.178570 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:30:38.178547 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07ba4cd4cb22a47b6d2e33529b3f9b93406c95c75bbc823cfd15d19c20a1f0dd\": container with ID starting with 07ba4cd4cb22a47b6d2e33529b3f9b93406c95c75bbc823cfd15d19c20a1f0dd not found: ID does not exist" containerID="07ba4cd4cb22a47b6d2e33529b3f9b93406c95c75bbc823cfd15d19c20a1f0dd" Sep 10 19:30:38.178657 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.178582 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07ba4cd4cb22a47b6d2e33529b3f9b93406c95c75bbc823cfd15d19c20a1f0dd"} err="failed to get container status \"07ba4cd4cb22a47b6d2e33529b3f9b93406c95c75bbc823cfd15d19c20a1f0dd\": rpc error: code = NotFound desc = could not find container \"07ba4cd4cb22a47b6d2e33529b3f9b93406c95c75bbc823cfd15d19c20a1f0dd\": container with ID starting with 07ba4cd4cb22a47b6d2e33529b3f9b93406c95c75bbc823cfd15d19c20a1f0dd not found: ID does not exist" Sep 10 19:30:38.180131 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.180112 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz"] Sep 10 19:30:38.184289 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.184270 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-588a50a1-5c585bb6fd-6tqtz"] Sep 10 19:30:38.196370 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.196339 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d"] Sep 10 19:30:38.206473 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:38.206418 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-588a50a1-7d6fc5794f-5xs8d"] Sep 10 19:30:39.390304 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:39.390270 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01a39a1a-130d-4ee9-a7a3-9c8822db3789" path="/var/lib/kubelet/pods/01a39a1a-130d-4ee9-a7a3-9c8822db3789/volumes" Sep 10 19:30:39.390672 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:39.390622 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="555799aa-0cf2-43dc-987e-53adaf87fdd2" path="/var/lib/kubelet/pods/555799aa-0cf2-43dc-987e-53adaf87fdd2/volumes" Sep 10 19:30:39.390985 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:30:39.390972 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f92f692d-999c-42dc-ba6d-2225ffd0dceb" path="/var/lib/kubelet/pods/f92f692d-999c-42dc-ba6d-2225ffd0dceb/volumes" Sep 10 19:31:04.982601 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.982569 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk"] Sep 10 19:31:04.984989 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.982936 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a39a1a-130d-4ee9-a7a3-9c8822db3789" containerName="payload-processing" Sep 10 19:31:04.984989 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.982949 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a39a1a-130d-4ee9-a7a3-9c8822db3789" containerName="payload-processing" Sep 10 19:31:04.984989 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.982960 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f92f692d-999c-42dc-ba6d-2225ffd0dceb" containerName="payload-pre-processing" Sep 10 19:31:04.984989 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.982966 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f92f692d-999c-42dc-ba6d-2225ffd0dceb" containerName="payload-pre-processing" Sep 10 19:31:04.984989 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.982986 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="555799aa-0cf2-43dc-987e-53adaf87fdd2" containerName="revoke-keys" Sep 10 19:31:04.984989 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.982992 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="555799aa-0cf2-43dc-987e-53adaf87fdd2" containerName="revoke-keys" Sep 10 19:31:04.984989 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.983043 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f92f692d-999c-42dc-ba6d-2225ffd0dceb" containerName="payload-pre-processing" Sep 10 19:31:04.984989 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.983050 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="01a39a1a-130d-4ee9-a7a3-9c8822db3789" containerName="payload-processing" Sep 10 19:31:04.984989 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.983058 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="555799aa-0cf2-43dc-987e-53adaf87fdd2" containerName="revoke-keys" Sep 10 19:31:04.985914 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.985895 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk" Sep 10 19:31:04.988093 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.988070 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:31:04.988093 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.988089 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkgt8\"" Sep 10 19:31:04.993316 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:04.993294 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk"] Sep 10 19:31:05.056539 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:05.056510 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7037376-eedf-45ab-8444-472f2574f6ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-faed1b-d2phk\" (UID: \"f7037376-eedf-45ab-8444-472f2574f6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk" Sep 10 19:31:05.157478 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:05.157448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7037376-eedf-45ab-8444-472f2574f6ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-faed1b-d2phk\" (UID: \"f7037376-eedf-45ab-8444-472f2574f6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk" Sep 10 19:31:05.158055 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:05.158036 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7037376-eedf-45ab-8444-472f2574f6ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-faed1b-d2phk\" (UID: \"f7037376-eedf-45ab-8444-472f2574f6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk" Sep 10 19:31:05.298688 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:05.298608 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk" Sep 10 19:31:05.574593 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:05.574568 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk"] Sep 10 19:31:05.576606 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:31:05.576570 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7037376_eedf_45ab_8444_472f2574f6ba.slice/crio-5356b50b231bfcb67c61ecc81d03fde4361710d9e2a6b48a5f7ae4b87bf3696a WatchSource:0}: Error finding container 5356b50b231bfcb67c61ecc81d03fde4361710d9e2a6b48a5f7ae4b87bf3696a: Status 404 returned error can't find the container with id 5356b50b231bfcb67c61ecc81d03fde4361710d9e2a6b48a5f7ae4b87bf3696a Sep 10 19:31:06.297716 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:06.297682 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk" event={"ID":"f7037376-eedf-45ab-8444-472f2574f6ba","Type":"ContainerStarted","Data":"ab39d4dfd85a96ca153d0ea19286cf0b074d78e596835ad2d0fffc965b495e6d"} Sep 10 19:31:06.298015 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:06.297792 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk" event={"ID":"f7037376-eedf-45ab-8444-472f2574f6ba","Type":"ContainerStarted","Data":"5356b50b231bfcb67c61ecc81d03fde4361710d9e2a6b48a5f7ae4b87bf3696a"} Sep 10 19:31:07.304358 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:07.304319 2578 generic.go:356] "Generic (PLEG): container finished" podID="f7037376-eedf-45ab-8444-472f2574f6ba" containerID="ab39d4dfd85a96ca153d0ea19286cf0b074d78e596835ad2d0fffc965b495e6d" exitCode=0 Sep 10 19:31:07.304801 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:07.304416 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk" event={"ID":"f7037376-eedf-45ab-8444-472f2574f6ba","Type":"ContainerDied","Data":"ab39d4dfd85a96ca153d0ea19286cf0b074d78e596835ad2d0fffc965b495e6d"} Sep 10 19:31:07.434344 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:07.434318 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk" Sep 10 19:31:07.479620 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:07.479592 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7037376-eedf-45ab-8444-472f2574f6ba-maas-api-service-ca\") pod \"f7037376-eedf-45ab-8444-472f2574f6ba\" (UID: \"f7037376-eedf-45ab-8444-472f2574f6ba\") " Sep 10 19:31:07.479967 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:07.479946 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7037376-eedf-45ab-8444-472f2574f6ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7037376-eedf-45ab-8444-472f2574f6ba" (UID: "f7037376-eedf-45ab-8444-472f2574f6ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:31:07.581145 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:07.581088 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7037376-eedf-45ab-8444-472f2574f6ba-maas-api-service-ca\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:31:08.310426 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:08.310391 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk" event={"ID":"f7037376-eedf-45ab-8444-472f2574f6ba","Type":"ContainerDied","Data":"5356b50b231bfcb67c61ecc81d03fde4361710d9e2a6b48a5f7ae4b87bf3696a"} Sep 10 19:31:08.310426 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:08.310426 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk" Sep 10 19:31:08.310880 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:08.310438 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5356b50b231bfcb67c61ecc81d03fde4361710d9e2a6b48a5f7ae4b87bf3696a" Sep 10 19:31:15.572994 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:15.572958 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44"] Sep 10 19:31:15.573469 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:15.573197 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" podUID="8a6d133c-2eb7-4005-aa50-932853269cbd" containerName="payload-processing" containerID="cri-o://987e427d363f1de6577d5a6ba2dc33007115cc76be72ba63383686ec8bd0beb5" gracePeriod=30 Sep 10 19:31:15.766184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:15.766125 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk"] Sep 10 19:31:15.769571 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:15.769495 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-faed1b-d2phk"] Sep 10 19:31:15.835534 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:15.835479 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" Sep 10 19:31:15.952850 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:15.952818 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6d133c-2eb7-4005-aa50-932853269cbd-plugins-config-volume\") pod \"8a6d133c-2eb7-4005-aa50-932853269cbd\" (UID: \"8a6d133c-2eb7-4005-aa50-932853269cbd\") " Sep 10 19:31:15.953012 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:15.952892 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85tzp\" (UniqueName: \"kubernetes.io/projected/8a6d133c-2eb7-4005-aa50-932853269cbd-kube-api-access-85tzp\") pod \"8a6d133c-2eb7-4005-aa50-932853269cbd\" (UID: \"8a6d133c-2eb7-4005-aa50-932853269cbd\") " Sep 10 19:31:15.953182 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:15.953156 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6d133c-2eb7-4005-aa50-932853269cbd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8a6d133c-2eb7-4005-aa50-932853269cbd" (UID: "8a6d133c-2eb7-4005-aa50-932853269cbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:31:15.955075 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:15.955050 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6d133c-2eb7-4005-aa50-932853269cbd-kube-api-access-85tzp" (OuterVolumeSpecName: "kube-api-access-85tzp") pod "8a6d133c-2eb7-4005-aa50-932853269cbd" (UID: "8a6d133c-2eb7-4005-aa50-932853269cbd"). InnerVolumeSpecName "kube-api-access-85tzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:31:16.053860 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:16.053828 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a6d133c-2eb7-4005-aa50-932853269cbd-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:31:16.053860 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:16.053855 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85tzp\" (UniqueName: \"kubernetes.io/projected/8a6d133c-2eb7-4005-aa50-932853269cbd-kube-api-access-85tzp\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:31:16.353057 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:16.353022 2578 generic.go:356] "Generic (PLEG): container finished" podID="8a6d133c-2eb7-4005-aa50-932853269cbd" containerID="987e427d363f1de6577d5a6ba2dc33007115cc76be72ba63383686ec8bd0beb5" exitCode=0 Sep 10 19:31:16.353218 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:16.353066 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" Sep 10 19:31:16.353218 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:16.353097 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" event={"ID":"8a6d133c-2eb7-4005-aa50-932853269cbd","Type":"ContainerDied","Data":"987e427d363f1de6577d5a6ba2dc33007115cc76be72ba63383686ec8bd0beb5"} Sep 10 19:31:16.353218 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:16.353142 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44" event={"ID":"8a6d133c-2eb7-4005-aa50-932853269cbd","Type":"ContainerDied","Data":"9282800bf9915f71bca8136f60211c098178aaafa2071dd48fc92477343f1d22"} Sep 10 19:31:16.353218 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:16.353166 2578 scope.go:117] "RemoveContainer" containerID="987e427d363f1de6577d5a6ba2dc33007115cc76be72ba63383686ec8bd0beb5" Sep 10 19:31:16.362133 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:16.362115 2578 scope.go:117] "RemoveContainer" containerID="987e427d363f1de6577d5a6ba2dc33007115cc76be72ba63383686ec8bd0beb5" Sep 10 19:31:16.362403 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:31:16.362387 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987e427d363f1de6577d5a6ba2dc33007115cc76be72ba63383686ec8bd0beb5\": container with ID starting with 987e427d363f1de6577d5a6ba2dc33007115cc76be72ba63383686ec8bd0beb5 not found: ID does not exist" containerID="987e427d363f1de6577d5a6ba2dc33007115cc76be72ba63383686ec8bd0beb5" Sep 10 19:31:16.362448 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:16.362412 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987e427d363f1de6577d5a6ba2dc33007115cc76be72ba63383686ec8bd0beb5"} err="failed to get container status \"987e427d363f1de6577d5a6ba2dc33007115cc76be72ba63383686ec8bd0beb5\": rpc error: code = NotFound desc = could not find container \"987e427d363f1de6577d5a6ba2dc33007115cc76be72ba63383686ec8bd0beb5\": container with ID starting with 987e427d363f1de6577d5a6ba2dc33007115cc76be72ba63383686ec8bd0beb5 not found: ID does not exist" Sep 10 19:31:16.373493 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:16.373470 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44"] Sep 10 19:31:16.377767 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:16.377747 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-faed1b-78cff8c469-n6n44"] Sep 10 19:31:17.390373 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:17.390332 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a6d133c-2eb7-4005-aa50-932853269cbd" path="/var/lib/kubelet/pods/8a6d133c-2eb7-4005-aa50-932853269cbd/volumes" Sep 10 19:31:17.390830 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:31:17.390644 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7037376-eedf-45ab-8444-472f2574f6ba" path="/var/lib/kubelet/pods/f7037376-eedf-45ab-8444-472f2574f6ba/volumes" Sep 10 19:32:05.328947 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.328905 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz"] Sep 10 19:32:05.329379 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.329237 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a6d133c-2eb7-4005-aa50-932853269cbd" containerName="payload-processing" Sep 10 19:32:05.329379 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.329248 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a6d133c-2eb7-4005-aa50-932853269cbd" containerName="payload-processing" Sep 10 19:32:05.329379 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.329266 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7037376-eedf-45ab-8444-472f2574f6ba" containerName="revoke-keys" Sep 10 19:32:05.329379 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.329271 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7037376-eedf-45ab-8444-472f2574f6ba" containerName="revoke-keys" Sep 10 19:32:05.329379 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.329325 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7037376-eedf-45ab-8444-472f2574f6ba" containerName="revoke-keys" Sep 10 19:32:05.329379 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.329333 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a6d133c-2eb7-4005-aa50-932853269cbd" containerName="payload-processing" Sep 10 19:32:05.333375 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.333360 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz" Sep 10 19:32:05.337322 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.337293 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkgt8\"" Sep 10 19:32:05.337434 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.337388 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:32:05.351453 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.351428 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz"] Sep 10 19:32:05.352083 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.352060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47082755-1f3c-4690-b146-77791e0eecbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz\" (UID: \"47082755-1f3c-4690-b146-77791e0eecbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz" Sep 10 19:32:05.452897 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.452863 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47082755-1f3c-4690-b146-77791e0eecbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz\" (UID: \"47082755-1f3c-4690-b146-77791e0eecbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz" Sep 10 19:32:05.453444 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.453423 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47082755-1f3c-4690-b146-77791e0eecbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz\" (UID: \"47082755-1f3c-4690-b146-77791e0eecbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz" Sep 10 19:32:05.642805 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.642766 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz" Sep 10 19:32:05.913749 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:05.913704 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz"] Sep 10 19:32:05.915831 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:32:05.915795 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47082755_1f3c_4690_b146_77791e0eecbe.slice/crio-671c43932a4e21d53069629716b0f78028379dfc41281c655c7d4237bdc9786e WatchSource:0}: Error finding container 671c43932a4e21d53069629716b0f78028379dfc41281c655c7d4237bdc9786e: Status 404 returned error can't find the container with id 671c43932a4e21d53069629716b0f78028379dfc41281c655c7d4237bdc9786e Sep 10 19:32:06.618899 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:06.618864 2578 generic.go:356] "Generic (PLEG): container finished" podID="47082755-1f3c-4690-b146-77791e0eecbe" containerID="9349504668c2eea8d5f2a7235579d0fc6a5092fa9cc3cb2620a255d56dabadc3" exitCode=0 Sep 10 19:32:06.619225 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:06.618959 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz" event={"ID":"47082755-1f3c-4690-b146-77791e0eecbe","Type":"ContainerDied","Data":"9349504668c2eea8d5f2a7235579d0fc6a5092fa9cc3cb2620a255d56dabadc3"} Sep 10 19:32:06.619225 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:06.619028 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz" event={"ID":"47082755-1f3c-4690-b146-77791e0eecbe","Type":"ContainerStarted","Data":"671c43932a4e21d53069629716b0f78028379dfc41281c655c7d4237bdc9786e"} Sep 10 19:32:07.754969 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:07.754945 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz" Sep 10 19:32:07.771089 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:07.771061 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47082755-1f3c-4690-b146-77791e0eecbe-maas-api-service-ca\") pod \"47082755-1f3c-4690-b146-77791e0eecbe\" (UID: \"47082755-1f3c-4690-b146-77791e0eecbe\") " Sep 10 19:32:07.771569 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:07.771543 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47082755-1f3c-4690-b146-77791e0eecbe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47082755-1f3c-4690-b146-77791e0eecbe" (UID: "47082755-1f3c-4690-b146-77791e0eecbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:32:07.871982 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:07.871951 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47082755-1f3c-4690-b146-77791e0eecbe-maas-api-service-ca\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:32:08.629987 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:08.629957 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz" Sep 10 19:32:08.630155 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:08.630006 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz" event={"ID":"47082755-1f3c-4690-b146-77791e0eecbe","Type":"ContainerDied","Data":"671c43932a4e21d53069629716b0f78028379dfc41281c655c7d4237bdc9786e"} Sep 10 19:32:08.630155 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:08.630061 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="671c43932a4e21d53069629716b0f78028379dfc41281c655c7d4237bdc9786e" Sep 10 19:32:16.149756 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.149666 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8"] Sep 10 19:32:16.150305 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.150267 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" podUID="f7bfd7b8-44cb-47f2-872d-0ebadfe0af30" containerName="payload-pre-processing" containerID="cri-o://608803ed4771799749562b199e25efcba80c85197b7578dcf9b5fe812d47868f" gracePeriod=30 Sep 10 19:32:16.331675 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.331639 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz"] Sep 10 19:32:16.333805 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.333782 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8300b9-7qztz"] Sep 10 19:32:16.413292 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.413236 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" Sep 10 19:32:16.447217 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.447177 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dkxb\" (UniqueName: \"kubernetes.io/projected/f7bfd7b8-44cb-47f2-872d-0ebadfe0af30-kube-api-access-9dkxb\") pod \"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30\" (UID: \"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30\") " Sep 10 19:32:16.447217 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.447216 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bfd7b8-44cb-47f2-872d-0ebadfe0af30-plugins-config-volume\") pod \"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30\" (UID: \"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30\") " Sep 10 19:32:16.447623 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.447596 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bfd7b8-44cb-47f2-872d-0ebadfe0af30-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7bfd7b8-44cb-47f2-872d-0ebadfe0af30" (UID: "f7bfd7b8-44cb-47f2-872d-0ebadfe0af30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:32:16.450016 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.449982 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bfd7b8-44cb-47f2-872d-0ebadfe0af30-kube-api-access-9dkxb" (OuterVolumeSpecName: "kube-api-access-9dkxb") pod "f7bfd7b8-44cb-47f2-872d-0ebadfe0af30" (UID: "f7bfd7b8-44cb-47f2-872d-0ebadfe0af30"). InnerVolumeSpecName "kube-api-access-9dkxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:32:16.548540 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.548510 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dkxb\" (UniqueName: \"kubernetes.io/projected/f7bfd7b8-44cb-47f2-872d-0ebadfe0af30-kube-api-access-9dkxb\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:32:16.548540 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.548540 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7bfd7b8-44cb-47f2-872d-0ebadfe0af30-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:32:16.670761 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.670668 2578 generic.go:356] "Generic (PLEG): container finished" podID="f7bfd7b8-44cb-47f2-872d-0ebadfe0af30" containerID="608803ed4771799749562b199e25efcba80c85197b7578dcf9b5fe812d47868f" exitCode=0 Sep 10 19:32:16.670761 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.670706 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" Sep 10 19:32:16.670942 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.670759 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" event={"ID":"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30","Type":"ContainerDied","Data":"608803ed4771799749562b199e25efcba80c85197b7578dcf9b5fe812d47868f"} Sep 10 19:32:16.670942 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.670816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8" event={"ID":"f7bfd7b8-44cb-47f2-872d-0ebadfe0af30","Type":"ContainerDied","Data":"bd5ff1f8e7279cfcea8da0c0ed799b1182c3fa57402744de3ed5bff8925623fc"} Sep 10 19:32:16.670942 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.670838 2578 scope.go:117] "RemoveContainer" containerID="608803ed4771799749562b199e25efcba80c85197b7578dcf9b5fe812d47868f" Sep 10 19:32:16.680216 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.680198 2578 scope.go:117] "RemoveContainer" containerID="608803ed4771799749562b199e25efcba80c85197b7578dcf9b5fe812d47868f" Sep 10 19:32:16.680489 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:32:16.680473 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608803ed4771799749562b199e25efcba80c85197b7578dcf9b5fe812d47868f\": container with ID starting with 608803ed4771799749562b199e25efcba80c85197b7578dcf9b5fe812d47868f not found: ID does not exist" containerID="608803ed4771799749562b199e25efcba80c85197b7578dcf9b5fe812d47868f" Sep 10 19:32:16.680538 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.680501 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608803ed4771799749562b199e25efcba80c85197b7578dcf9b5fe812d47868f"} err="failed to get container status \"608803ed4771799749562b199e25efcba80c85197b7578dcf9b5fe812d47868f\": rpc error: code = NotFound desc = could not find container \"608803ed4771799749562b199e25efcba80c85197b7578dcf9b5fe812d47868f\": container with ID starting with 608803ed4771799749562b199e25efcba80c85197b7578dcf9b5fe812d47868f not found: ID does not exist" Sep 10 19:32:16.692712 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.692688 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8"] Sep 10 19:32:16.696657 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:16.696635 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-8300b9-5474bb45f4-xv6f8"] Sep 10 19:32:17.390954 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:17.390918 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47082755-1f3c-4690-b146-77791e0eecbe" path="/var/lib/kubelet/pods/47082755-1f3c-4690-b146-77791e0eecbe/volumes" Sep 10 19:32:17.391395 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:17.391379 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7bfd7b8-44cb-47f2-872d-0ebadfe0af30" path="/var/lib/kubelet/pods/f7bfd7b8-44cb-47f2-872d-0ebadfe0af30/volumes" Sep 10 19:32:21.007753 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.007699 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9"] Sep 10 19:32:21.008101 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.008052 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7bfd7b8-44cb-47f2-872d-0ebadfe0af30" containerName="payload-pre-processing" Sep 10 19:32:21.008101 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.008064 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7bfd7b8-44cb-47f2-872d-0ebadfe0af30" containerName="payload-pre-processing" Sep 10 19:32:21.008101 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.008071 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47082755-1f3c-4690-b146-77791e0eecbe" containerName="revoke-keys" Sep 10 19:32:21.008101 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.008076 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="47082755-1f3c-4690-b146-77791e0eecbe" containerName="revoke-keys" Sep 10 19:32:21.008262 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.008134 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="47082755-1f3c-4690-b146-77791e0eecbe" containerName="revoke-keys" Sep 10 19:32:21.008262 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.008142 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7bfd7b8-44cb-47f2-872d-0ebadfe0af30" containerName="payload-pre-processing" Sep 10 19:32:21.012087 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.012068 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" Sep 10 19:32:21.014740 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.014699 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9d2e2715\"" Sep 10 19:32:21.024003 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.023979 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9"] Sep 10 19:32:21.086456 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.086424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlnjc\" (UniqueName: \"kubernetes.io/projected/17efc055-99f4-4796-abf4-c44b1d54d94b-kube-api-access-tlnjc\") pod \"payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9\" (UID: \"17efc055-99f4-4796-abf4-c44b1d54d94b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" Sep 10 19:32:21.086604 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.086477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17efc055-99f4-4796-abf4-c44b1d54d94b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9\" (UID: \"17efc055-99f4-4796-abf4-c44b1d54d94b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" Sep 10 19:32:21.187551 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.187510 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlnjc\" (UniqueName: \"kubernetes.io/projected/17efc055-99f4-4796-abf4-c44b1d54d94b-kube-api-access-tlnjc\") pod \"payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9\" (UID: \"17efc055-99f4-4796-abf4-c44b1d54d94b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" Sep 10 19:32:21.187749 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.187570 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17efc055-99f4-4796-abf4-c44b1d54d94b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9\" (UID: \"17efc055-99f4-4796-abf4-c44b1d54d94b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" Sep 10 19:32:21.188260 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.188234 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17efc055-99f4-4796-abf4-c44b1d54d94b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9\" (UID: \"17efc055-99f4-4796-abf4-c44b1d54d94b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" Sep 10 19:32:21.195677 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.195647 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlnjc\" (UniqueName: \"kubernetes.io/projected/17efc055-99f4-4796-abf4-c44b1d54d94b-kube-api-access-tlnjc\") pod \"payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9\" (UID: \"17efc055-99f4-4796-abf4-c44b1d54d94b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" Sep 10 19:32:21.322054 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.321987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" Sep 10 19:32:21.599840 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.599819 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9"] Sep 10 19:32:21.601900 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:32:21.601872 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17efc055_99f4_4796_abf4_c44b1d54d94b.slice/crio-df26285b0b0f1916e265266a56c9ac8b861a33df9df0479599ddcad8363d25bc WatchSource:0}: Error finding container df26285b0b0f1916e265266a56c9ac8b861a33df9df0479599ddcad8363d25bc: Status 404 returned error can't find the container with id df26285b0b0f1916e265266a56c9ac8b861a33df9df0479599ddcad8363d25bc Sep 10 19:32:21.697667 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.697623 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" event={"ID":"17efc055-99f4-4796-abf4-c44b1d54d94b","Type":"ContainerStarted","Data":"01b19711e56816c20cdcb8b7b9d76f4e867f93efdb60066588d402fb5b8ed635"} Sep 10 19:32:21.697837 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.697696 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" event={"ID":"17efc055-99f4-4796-abf4-c44b1d54d94b","Type":"ContainerStarted","Data":"df26285b0b0f1916e265266a56c9ac8b861a33df9df0479599ddcad8363d25bc"} Sep 10 19:32:21.697837 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.697760 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" Sep 10 19:32:21.728865 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:21.728814 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" podStartSLOduration=1.7287978499999999 podStartE2EDuration="1.72879785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:21.727005849 +0000 UTC m=+990.857507477" watchObservedRunningTime="2026-09-10 19:32:21.72879785 +0000 UTC m=+990.859299481" Sep 10 19:32:23.638979 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:23.638942 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722"] Sep 10 19:32:23.642402 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:23.642385 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722" Sep 10 19:32:23.644556 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:23.644531 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:32:23.644671 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:23.644557 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkgt8\"" Sep 10 19:32:23.648468 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:23.648444 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722"] Sep 10 19:32:23.709154 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:23.709124 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9beddc-80ca-4a64-93ce-5449077c9b6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4247c7-h9722\" (UID: \"2c9beddc-80ca-4a64-93ce-5449077c9b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722" Sep 10 19:32:23.809825 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:23.809797 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9beddc-80ca-4a64-93ce-5449077c9b6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4247c7-h9722\" (UID: \"2c9beddc-80ca-4a64-93ce-5449077c9b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722" Sep 10 19:32:23.810521 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:23.810498 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9beddc-80ca-4a64-93ce-5449077c9b6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4247c7-h9722\" (UID: \"2c9beddc-80ca-4a64-93ce-5449077c9b6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722" Sep 10 19:32:23.954766 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:23.954658 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722" Sep 10 19:32:24.229714 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:24.229691 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722"] Sep 10 19:32:24.231373 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:32:24.231347 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9beddc_80ca_4a64_93ce_5449077c9b6d.slice/crio-b1ca93347eeb3e04eeaca51353a2fd5d6f6cbb8b294ce32aad15114c63a9ae55 WatchSource:0}: Error finding container b1ca93347eeb3e04eeaca51353a2fd5d6f6cbb8b294ce32aad15114c63a9ae55: Status 404 returned error can't find the container with id b1ca93347eeb3e04eeaca51353a2fd5d6f6cbb8b294ce32aad15114c63a9ae55 Sep 10 19:32:24.714131 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:24.714088 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722" event={"ID":"2c9beddc-80ca-4a64-93ce-5449077c9b6d","Type":"ContainerStarted","Data":"abfb23b00f0bf8cb691bc2329c4e45fbbc1e710cf03b509db958bf7bbc401f2a"} Sep 10 19:32:24.714531 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:24.714147 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722" event={"ID":"2c9beddc-80ca-4a64-93ce-5449077c9b6d","Type":"ContainerStarted","Data":"b1ca93347eeb3e04eeaca51353a2fd5d6f6cbb8b294ce32aad15114c63a9ae55"} Sep 10 19:32:24.729404 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:24.729356 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722" podStartSLOduration=1.7293428990000002 podStartE2EDuration="1.729342899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:24.727579252 +0000 UTC m=+993.858080883" watchObservedRunningTime="2026-09-10 19:32:24.729342899 +0000 UTC m=+993.859844570" Sep 10 19:32:25.720048 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:25.720007 2578 generic.go:356] "Generic (PLEG): container finished" podID="2c9beddc-80ca-4a64-93ce-5449077c9b6d" containerID="abfb23b00f0bf8cb691bc2329c4e45fbbc1e710cf03b509db958bf7bbc401f2a" exitCode=0 Sep 10 19:32:25.720411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:25.720092 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722" event={"ID":"2c9beddc-80ca-4a64-93ce-5449077c9b6d","Type":"ContainerDied","Data":"abfb23b00f0bf8cb691bc2329c4e45fbbc1e710cf03b509db958bf7bbc401f2a"} Sep 10 19:32:26.851568 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:26.851539 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722" Sep 10 19:32:26.935397 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:26.935365 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9beddc-80ca-4a64-93ce-5449077c9b6d-maas-api-service-ca\") pod \"2c9beddc-80ca-4a64-93ce-5449077c9b6d\" (UID: \"2c9beddc-80ca-4a64-93ce-5449077c9b6d\") " Sep 10 19:32:26.935749 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:26.935703 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9beddc-80ca-4a64-93ce-5449077c9b6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c9beddc-80ca-4a64-93ce-5449077c9b6d" (UID: "2c9beddc-80ca-4a64-93ce-5449077c9b6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:32:27.036540 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:27.036458 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9beddc-80ca-4a64-93ce-5449077c9b6d-maas-api-service-ca\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:32:27.730507 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:27.730435 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722" event={"ID":"2c9beddc-80ca-4a64-93ce-5449077c9b6d","Type":"ContainerDied","Data":"b1ca93347eeb3e04eeaca51353a2fd5d6f6cbb8b294ce32aad15114c63a9ae55"} Sep 10 19:32:27.730507 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:27.730462 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722" Sep 10 19:32:27.730507 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:27.730482 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1ca93347eeb3e04eeaca51353a2fd5d6f6cbb8b294ce32aad15114c63a9ae55" Sep 10 19:32:32.705649 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:32.705619 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" Sep 10 19:32:34.360536 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.360498 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc"] Sep 10 19:32:34.364810 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.364786 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4247c7-29817810-f8ffc"] Sep 10 19:32:34.389706 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.389671 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn"] Sep 10 19:32:34.389998 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.389948 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" podUID="8671a993-6178-4d76-8a53-8d49ce48bc40" containerName="payload-processing" containerID="cri-o://6d1e96e681c4cc57f3ef371ead08a30c3e62bd147b279ee8e910b25655ace1d6" gracePeriod=30 Sep 10 19:32:34.397501 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.397278 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl"] Sep 10 19:32:34.397609 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.397521 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" podUID="7660a028-b331-4459-bbe6-2e0d913bc063" containerName="payload-pre-processing" containerID="cri-o://e55953b8fc3a7af8752f596e8e06366079a8a13b205c56afcbd2df6a95077fe6" gracePeriod=30 Sep 10 19:32:34.651598 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.651575 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" Sep 10 19:32:34.679420 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.679396 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" Sep 10 19:32:34.682746 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.682708 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722"] Sep 10 19:32:34.688569 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.688547 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4247c7-h9722"] Sep 10 19:32:34.701243 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.701217 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8671a993-6178-4d76-8a53-8d49ce48bc40-plugins-config-volume\") pod \"8671a993-6178-4d76-8a53-8d49ce48bc40\" (UID: \"8671a993-6178-4d76-8a53-8d49ce48bc40\") " Sep 10 19:32:34.701368 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.701253 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8xsn\" (UniqueName: \"kubernetes.io/projected/8671a993-6178-4d76-8a53-8d49ce48bc40-kube-api-access-m8xsn\") pod \"8671a993-6178-4d76-8a53-8d49ce48bc40\" (UID: \"8671a993-6178-4d76-8a53-8d49ce48bc40\") " Sep 10 19:32:34.701702 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.701667 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8671a993-6178-4d76-8a53-8d49ce48bc40-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8671a993-6178-4d76-8a53-8d49ce48bc40" (UID: "8671a993-6178-4d76-8a53-8d49ce48bc40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:32:34.703502 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.703476 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8671a993-6178-4d76-8a53-8d49ce48bc40-kube-api-access-m8xsn" (OuterVolumeSpecName: "kube-api-access-m8xsn") pod "8671a993-6178-4d76-8a53-8d49ce48bc40" (UID: "8671a993-6178-4d76-8a53-8d49ce48bc40"). InnerVolumeSpecName "kube-api-access-m8xsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:32:34.764257 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.764214 2578 generic.go:356] "Generic (PLEG): container finished" podID="7660a028-b331-4459-bbe6-2e0d913bc063" containerID="e55953b8fc3a7af8752f596e8e06366079a8a13b205c56afcbd2df6a95077fe6" exitCode=0 Sep 10 19:32:34.764447 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.764260 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" Sep 10 19:32:34.764447 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.764296 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" event={"ID":"7660a028-b331-4459-bbe6-2e0d913bc063","Type":"ContainerDied","Data":"e55953b8fc3a7af8752f596e8e06366079a8a13b205c56afcbd2df6a95077fe6"} Sep 10 19:32:34.764447 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.764363 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl" event={"ID":"7660a028-b331-4459-bbe6-2e0d913bc063","Type":"ContainerDied","Data":"7244c89c1b5cbecf6ac46f35695bcc4d74a0c4f29fa8ef40467ca0fda1b4cf1e"} Sep 10 19:32:34.764447 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.764393 2578 scope.go:117] "RemoveContainer" containerID="e55953b8fc3a7af8752f596e8e06366079a8a13b205c56afcbd2df6a95077fe6" Sep 10 19:32:34.765663 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.765641 2578 generic.go:356] "Generic (PLEG): container finished" podID="8671a993-6178-4d76-8a53-8d49ce48bc40" containerID="6d1e96e681c4cc57f3ef371ead08a30c3e62bd147b279ee8e910b25655ace1d6" exitCode=0 Sep 10 19:32:34.765814 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.765675 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" event={"ID":"8671a993-6178-4d76-8a53-8d49ce48bc40","Type":"ContainerDied","Data":"6d1e96e681c4cc57f3ef371ead08a30c3e62bd147b279ee8e910b25655ace1d6"} Sep 10 19:32:34.765814 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.765690 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" Sep 10 19:32:34.765921 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.765744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn" event={"ID":"8671a993-6178-4d76-8a53-8d49ce48bc40","Type":"ContainerDied","Data":"dc64c6f5e621319472859f478b913ffc8819d8af7d05e002e5b8ee7a055f2598"} Sep 10 19:32:34.776283 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.775717 2578 scope.go:117] "RemoveContainer" containerID="e55953b8fc3a7af8752f596e8e06366079a8a13b205c56afcbd2df6a95077fe6" Sep 10 19:32:34.777201 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:32:34.777172 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e55953b8fc3a7af8752f596e8e06366079a8a13b205c56afcbd2df6a95077fe6\": container with ID starting with e55953b8fc3a7af8752f596e8e06366079a8a13b205c56afcbd2df6a95077fe6 not found: ID does not exist" containerID="e55953b8fc3a7af8752f596e8e06366079a8a13b205c56afcbd2df6a95077fe6" Sep 10 19:32:34.777313 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.777212 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e55953b8fc3a7af8752f596e8e06366079a8a13b205c56afcbd2df6a95077fe6"} err="failed to get container status \"e55953b8fc3a7af8752f596e8e06366079a8a13b205c56afcbd2df6a95077fe6\": rpc error: code = NotFound desc = could not find container \"e55953b8fc3a7af8752f596e8e06366079a8a13b205c56afcbd2df6a95077fe6\": container with ID starting with e55953b8fc3a7af8752f596e8e06366079a8a13b205c56afcbd2df6a95077fe6 not found: ID does not exist" Sep 10 19:32:34.777313 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.777238 2578 scope.go:117] "RemoveContainer" containerID="6d1e96e681c4cc57f3ef371ead08a30c3e62bd147b279ee8e910b25655ace1d6" Sep 10 19:32:34.788363 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.788344 2578 scope.go:117] "RemoveContainer" containerID="6d1e96e681c4cc57f3ef371ead08a30c3e62bd147b279ee8e910b25655ace1d6" Sep 10 19:32:34.788684 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:32:34.788667 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1e96e681c4cc57f3ef371ead08a30c3e62bd147b279ee8e910b25655ace1d6\": container with ID starting with 6d1e96e681c4cc57f3ef371ead08a30c3e62bd147b279ee8e910b25655ace1d6 not found: ID does not exist" containerID="6d1e96e681c4cc57f3ef371ead08a30c3e62bd147b279ee8e910b25655ace1d6" Sep 10 19:32:34.788779 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.788692 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1e96e681c4cc57f3ef371ead08a30c3e62bd147b279ee8e910b25655ace1d6"} err="failed to get container status \"6d1e96e681c4cc57f3ef371ead08a30c3e62bd147b279ee8e910b25655ace1d6\": rpc error: code = NotFound desc = could not find container \"6d1e96e681c4cc57f3ef371ead08a30c3e62bd147b279ee8e910b25655ace1d6\": container with ID starting with 6d1e96e681c4cc57f3ef371ead08a30c3e62bd147b279ee8e910b25655ace1d6 not found: ID does not exist" Sep 10 19:32:34.789304 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.789286 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn"] Sep 10 19:32:34.794048 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.794014 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4247c7-869768d85c-54wcn"] Sep 10 19:32:34.802237 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.802213 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cn6zf\" (UniqueName: \"kubernetes.io/projected/7660a028-b331-4459-bbe6-2e0d913bc063-kube-api-access-cn6zf\") pod \"7660a028-b331-4459-bbe6-2e0d913bc063\" (UID: \"7660a028-b331-4459-bbe6-2e0d913bc063\") " Sep 10 19:32:34.802411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.802332 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7660a028-b331-4459-bbe6-2e0d913bc063-plugins-config-volume\") pod \"7660a028-b331-4459-bbe6-2e0d913bc063\" (UID: \"7660a028-b331-4459-bbe6-2e0d913bc063\") " Sep 10 19:32:34.802645 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.802622 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8671a993-6178-4d76-8a53-8d49ce48bc40-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:32:34.802645 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.802638 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8xsn\" (UniqueName: \"kubernetes.io/projected/8671a993-6178-4d76-8a53-8d49ce48bc40-kube-api-access-m8xsn\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:32:34.802803 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.802658 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7660a028-b331-4459-bbe6-2e0d913bc063-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7660a028-b331-4459-bbe6-2e0d913bc063" (UID: "7660a028-b331-4459-bbe6-2e0d913bc063"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:32:34.804355 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.804333 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7660a028-b331-4459-bbe6-2e0d913bc063-kube-api-access-cn6zf" (OuterVolumeSpecName: "kube-api-access-cn6zf") pod "7660a028-b331-4459-bbe6-2e0d913bc063" (UID: "7660a028-b331-4459-bbe6-2e0d913bc063"). InnerVolumeSpecName "kube-api-access-cn6zf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:32:34.903820 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.903759 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7660a028-b331-4459-bbe6-2e0d913bc063-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:32:34.903820 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:34.903780 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn6zf\" (UniqueName: \"kubernetes.io/projected/7660a028-b331-4459-bbe6-2e0d913bc063-kube-api-access-cn6zf\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:32:35.087676 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:35.087648 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl"] Sep 10 19:32:35.091415 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:35.091393 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4247c7-675f8c4966-p2hnl"] Sep 10 19:32:35.390650 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:35.390621 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c9beddc-80ca-4a64-93ce-5449077c9b6d" path="/var/lib/kubelet/pods/2c9beddc-80ca-4a64-93ce-5449077c9b6d/volumes" Sep 10 19:32:35.391034 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:35.391001 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7660a028-b331-4459-bbe6-2e0d913bc063" path="/var/lib/kubelet/pods/7660a028-b331-4459-bbe6-2e0d913bc063/volumes" Sep 10 19:32:35.391300 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:35.391287 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8671a993-6178-4d76-8a53-8d49ce48bc40" path="/var/lib/kubelet/pods/8671a993-6178-4d76-8a53-8d49ce48bc40/volumes" Sep 10 19:32:35.391516 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:35.391506 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7a7dbb0-aff0-4c57-8650-41d819bea794" path="/var/lib/kubelet/pods/a7a7dbb0-aff0-4c57-8650-41d819bea794/volumes" Sep 10 19:32:47.267004 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.266935 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw"] Sep 10 19:32:47.267411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.267257 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7660a028-b331-4459-bbe6-2e0d913bc063" containerName="payload-pre-processing" Sep 10 19:32:47.267411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.267269 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7660a028-b331-4459-bbe6-2e0d913bc063" containerName="payload-pre-processing" Sep 10 19:32:47.267411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.267280 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c9beddc-80ca-4a64-93ce-5449077c9b6d" containerName="revoke-keys" Sep 10 19:32:47.267411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.267285 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c9beddc-80ca-4a64-93ce-5449077c9b6d" containerName="revoke-keys" Sep 10 19:32:47.267411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.267298 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8671a993-6178-4d76-8a53-8d49ce48bc40" containerName="payload-processing" Sep 10 19:32:47.267411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.267303 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8671a993-6178-4d76-8a53-8d49ce48bc40" containerName="payload-processing" Sep 10 19:32:47.267411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.267356 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8671a993-6178-4d76-8a53-8d49ce48bc40" containerName="payload-processing" Sep 10 19:32:47.267411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.267364 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7660a028-b331-4459-bbe6-2e0d913bc063" containerName="payload-pre-processing" Sep 10 19:32:47.267411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.267370 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c9beddc-80ca-4a64-93ce-5449077c9b6d" containerName="revoke-keys" Sep 10 19:32:47.270140 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.270124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" Sep 10 19:32:47.272363 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.272344 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-14a8d1-dockercfg-wwkqr\"" Sep 10 19:32:47.272761 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.272745 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-14a8d1\"" Sep 10 19:32:47.283860 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.283841 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw"] Sep 10 19:32:47.334855 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.334828 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6"] Sep 10 19:32:47.338055 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.338035 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" Sep 10 19:32:47.348484 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.348457 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6"] Sep 10 19:32:47.410087 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.410058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jwvl\" (UniqueName: \"kubernetes.io/projected/32b343b4-2960-44b5-b701-b6f94ea5c181-kube-api-access-6jwvl\") pod \"payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw\" (UID: \"32b343b4-2960-44b5-b701-b6f94ea5c181\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" Sep 10 19:32:47.410087 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.410088 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207211f-1f91-4a49-99c3-d443939a4471-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6\" (UID: \"4207211f-1f91-4a49-99c3-d443939a4471\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" Sep 10 19:32:47.410272 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.410136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5cgx\" (UniqueName: \"kubernetes.io/projected/4207211f-1f91-4a49-99c3-d443939a4471-kube-api-access-s5cgx\") pod \"payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6\" (UID: \"4207211f-1f91-4a49-99c3-d443939a4471\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" Sep 10 19:32:47.410272 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.410184 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b343b4-2960-44b5-b701-b6f94ea5c181-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw\" (UID: \"32b343b4-2960-44b5-b701-b6f94ea5c181\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" Sep 10 19:32:47.510815 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.510788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b343b4-2960-44b5-b701-b6f94ea5c181-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw\" (UID: \"32b343b4-2960-44b5-b701-b6f94ea5c181\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" Sep 10 19:32:47.510953 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.510843 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jwvl\" (UniqueName: \"kubernetes.io/projected/32b343b4-2960-44b5-b701-b6f94ea5c181-kube-api-access-6jwvl\") pod \"payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw\" (UID: \"32b343b4-2960-44b5-b701-b6f94ea5c181\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" Sep 10 19:32:47.510953 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.510874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207211f-1f91-4a49-99c3-d443939a4471-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6\" (UID: \"4207211f-1f91-4a49-99c3-d443939a4471\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" Sep 10 19:32:47.510953 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.510937 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5cgx\" (UniqueName: \"kubernetes.io/projected/4207211f-1f91-4a49-99c3-d443939a4471-kube-api-access-s5cgx\") pod \"payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6\" (UID: \"4207211f-1f91-4a49-99c3-d443939a4471\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" Sep 10 19:32:47.511479 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.511459 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b343b4-2960-44b5-b701-b6f94ea5c181-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw\" (UID: \"32b343b4-2960-44b5-b701-b6f94ea5c181\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" Sep 10 19:32:47.511518 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.511469 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207211f-1f91-4a49-99c3-d443939a4471-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6\" (UID: \"4207211f-1f91-4a49-99c3-d443939a4471\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" Sep 10 19:32:47.520304 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.520247 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jwvl\" (UniqueName: \"kubernetes.io/projected/32b343b4-2960-44b5-b701-b6f94ea5c181-kube-api-access-6jwvl\") pod \"payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw\" (UID: \"32b343b4-2960-44b5-b701-b6f94ea5c181\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" Sep 10 19:32:47.520304 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.520278 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5cgx\" (UniqueName: \"kubernetes.io/projected/4207211f-1f91-4a49-99c3-d443939a4471-kube-api-access-s5cgx\") pod \"payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6\" (UID: \"4207211f-1f91-4a49-99c3-d443939a4471\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" Sep 10 19:32:47.580233 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.580207 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" Sep 10 19:32:47.648711 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.648683 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" Sep 10 19:32:47.852973 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.852946 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw"] Sep 10 19:32:47.855354 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:32:47.855329 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b343b4_2960_44b5_b701_b6f94ea5c181.slice/crio-1e9aba64383851df21941728efd0f1e9af8e4645f19d594280d388fffac8ee8c WatchSource:0}: Error finding container 1e9aba64383851df21941728efd0f1e9af8e4645f19d594280d388fffac8ee8c: Status 404 returned error can't find the container with id 1e9aba64383851df21941728efd0f1e9af8e4645f19d594280d388fffac8ee8c Sep 10 19:32:47.926745 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:47.926705 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6"] Sep 10 19:32:47.928059 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:32:47.928024 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4207211f_1f91_4a49_99c3_d443939a4471.slice/crio-975e053e5de4f49285d8801c04474d753757462bf6636feefc936aaa1e42e1de WatchSource:0}: Error finding container 975e053e5de4f49285d8801c04474d753757462bf6636feefc936aaa1e42e1de: Status 404 returned error can't find the container with id 975e053e5de4f49285d8801c04474d753757462bf6636feefc936aaa1e42e1de Sep 10 19:32:48.843867 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:48.843829 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" event={"ID":"32b343b4-2960-44b5-b701-b6f94ea5c181","Type":"ContainerStarted","Data":"ed89d7880d12fcbc6bcf31e75586334837653f44b5b9bd3d8b0fff944903bbe0"} Sep 10 19:32:48.844288 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:48.843889 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" event={"ID":"32b343b4-2960-44b5-b701-b6f94ea5c181","Type":"ContainerStarted","Data":"1e9aba64383851df21941728efd0f1e9af8e4645f19d594280d388fffac8ee8c"} Sep 10 19:32:48.844288 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:48.843918 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" Sep 10 19:32:48.845461 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:48.845432 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" event={"ID":"4207211f-1f91-4a49-99c3-d443939a4471","Type":"ContainerStarted","Data":"02bc402fb5a9812755b6a4cfc6adec4eca34b915a3749bc06ec86e9bae2ea17b"} Sep 10 19:32:48.845580 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:48.845476 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" event={"ID":"4207211f-1f91-4a49-99c3-d443939a4471","Type":"ContainerStarted","Data":"975e053e5de4f49285d8801c04474d753757462bf6636feefc936aaa1e42e1de"} Sep 10 19:32:48.845580 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:48.845497 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" Sep 10 19:32:48.860194 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:48.860150 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" podStartSLOduration=1.860134924 podStartE2EDuration="1.860134924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:48.859532588 +0000 UTC m=+1017.990034227" watchObservedRunningTime="2026-09-10 19:32:48.860134924 +0000 UTC m=+1017.990636554" Sep 10 19:32:48.878167 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:48.878122 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" podStartSLOduration=1.87810701 podStartE2EDuration="1.87810701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:32:48.876539156 +0000 UTC m=+1018.007040788" watchObservedRunningTime="2026-09-10 19:32:48.87810701 +0000 UTC m=+1018.008608641" Sep 10 19:32:59.852920 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:59.852888 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" Sep 10 19:32:59.853437 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:32:59.853044 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" Sep 10 19:33:02.531990 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.531957 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9"] Sep 10 19:33:02.532452 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.532233 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" podUID="17efc055-99f4-4796-abf4-c44b1d54d94b" containerName="payload-pre-processing" containerID="cri-o://01b19711e56816c20cdcb8b7b9d76f4e867f93efdb60066588d402fb5b8ed635" gracePeriod=30 Sep 10 19:33:02.776029 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.776005 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" Sep 10 19:33:02.840612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.840518 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlnjc\" (UniqueName: \"kubernetes.io/projected/17efc055-99f4-4796-abf4-c44b1d54d94b-kube-api-access-tlnjc\") pod \"17efc055-99f4-4796-abf4-c44b1d54d94b\" (UID: \"17efc055-99f4-4796-abf4-c44b1d54d94b\") " Sep 10 19:33:02.840612 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.840572 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17efc055-99f4-4796-abf4-c44b1d54d94b-plugins-config-volume\") pod \"17efc055-99f4-4796-abf4-c44b1d54d94b\" (UID: \"17efc055-99f4-4796-abf4-c44b1d54d94b\") " Sep 10 19:33:02.841018 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.840985 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17efc055-99f4-4796-abf4-c44b1d54d94b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17efc055-99f4-4796-abf4-c44b1d54d94b" (UID: "17efc055-99f4-4796-abf4-c44b1d54d94b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:33:02.842777 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.842752 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17efc055-99f4-4796-abf4-c44b1d54d94b-kube-api-access-tlnjc" (OuterVolumeSpecName: "kube-api-access-tlnjc") pod "17efc055-99f4-4796-abf4-c44b1d54d94b" (UID: "17efc055-99f4-4796-abf4-c44b1d54d94b"). InnerVolumeSpecName "kube-api-access-tlnjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:33:02.914814 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.914778 2578 generic.go:356] "Generic (PLEG): container finished" podID="17efc055-99f4-4796-abf4-c44b1d54d94b" containerID="01b19711e56816c20cdcb8b7b9d76f4e867f93efdb60066588d402fb5b8ed635" exitCode=0 Sep 10 19:33:02.914949 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.914821 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" Sep 10 19:33:02.914949 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.914874 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" event={"ID":"17efc055-99f4-4796-abf4-c44b1d54d94b","Type":"ContainerDied","Data":"01b19711e56816c20cdcb8b7b9d76f4e867f93efdb60066588d402fb5b8ed635"} Sep 10 19:33:02.914949 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.914921 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" event={"ID":"17efc055-99f4-4796-abf4-c44b1d54d94b","Type":"ContainerDied","Data":"df26285b0b0f1916e265266a56c9ac8b861a33df9df0479599ddcad8363d25bc"} Sep 10 19:33:02.914949 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.914944 2578 scope.go:117] "RemoveContainer" containerID="01b19711e56816c20cdcb8b7b9d76f4e867f93efdb60066588d402fb5b8ed635" Sep 10 19:33:02.928855 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.928836 2578 scope.go:117] "RemoveContainer" containerID="01b19711e56816c20cdcb8b7b9d76f4e867f93efdb60066588d402fb5b8ed635" Sep 10 19:33:02.929113 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:33:02.929089 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b19711e56816c20cdcb8b7b9d76f4e867f93efdb60066588d402fb5b8ed635\": container with ID starting with 01b19711e56816c20cdcb8b7b9d76f4e867f93efdb60066588d402fb5b8ed635 not found: ID does not exist" containerID="01b19711e56816c20cdcb8b7b9d76f4e867f93efdb60066588d402fb5b8ed635" Sep 10 19:33:02.929164 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.929121 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b19711e56816c20cdcb8b7b9d76f4e867f93efdb60066588d402fb5b8ed635"} err="failed to get container status \"01b19711e56816c20cdcb8b7b9d76f4e867f93efdb60066588d402fb5b8ed635\": rpc error: code = NotFound desc = could not find container \"01b19711e56816c20cdcb8b7b9d76f4e867f93efdb60066588d402fb5b8ed635\": container with ID starting with 01b19711e56816c20cdcb8b7b9d76f4e867f93efdb60066588d402fb5b8ed635 not found: ID does not exist" Sep 10 19:33:02.938918 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.938897 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9"] Sep 10 19:33:02.941295 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.941270 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlnjc\" (UniqueName: \"kubernetes.io/projected/17efc055-99f4-4796-abf4-c44b1d54d94b-kube-api-access-tlnjc\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:33:02.941420 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.941298 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17efc055-99f4-4796-abf4-c44b1d54d94b-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:33:02.942847 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:02.942828 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9"] Sep 10 19:33:03.391187 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:03.391156 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17efc055-99f4-4796-abf4-c44b1d54d94b" path="/var/lib/kubelet/pods/17efc055-99f4-4796-abf4-c44b1d54d94b/volumes" Sep 10 19:33:03.704163 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:03.704071 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-9d2e2715-dc85b9ff4-8thc9" podUID="17efc055-99f4-4796-abf4-c44b1d54d94b" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.48:9004: i/o timeout" Sep 10 19:33:15.302198 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.302166 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz"] Sep 10 19:33:15.302587 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.302516 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17efc055-99f4-4796-abf4-c44b1d54d94b" containerName="payload-pre-processing" Sep 10 19:33:15.302587 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.302528 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="17efc055-99f4-4796-abf4-c44b1d54d94b" containerName="payload-pre-processing" Sep 10 19:33:15.302587 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.302584 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="17efc055-99f4-4796-abf4-c44b1d54d94b" containerName="payload-pre-processing" Sep 10 19:33:15.306636 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.306608 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" Sep 10 19:33:15.308824 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.308803 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-cab4ee-dockercfg-cpcgg\"" Sep 10 19:33:15.308945 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.308807 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-cab4ee\"" Sep 10 19:33:15.316699 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.316666 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz"] Sep 10 19:33:15.342485 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.342461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nltzz\" (UniqueName: \"kubernetes.io/projected/db04029d-705d-459d-b42d-03df6c5f845e-kube-api-access-nltzz\") pod \"payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz\" (UID: \"db04029d-705d-459d-b42d-03df6c5f845e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" Sep 10 19:33:15.342579 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.342530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db04029d-705d-459d-b42d-03df6c5f845e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz\" (UID: \"db04029d-705d-459d-b42d-03df6c5f845e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" Sep 10 19:33:15.443358 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.443327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db04029d-705d-459d-b42d-03df6c5f845e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz\" (UID: \"db04029d-705d-459d-b42d-03df6c5f845e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" Sep 10 19:33:15.443524 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.443505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nltzz\" (UniqueName: \"kubernetes.io/projected/db04029d-705d-459d-b42d-03df6c5f845e-kube-api-access-nltzz\") pod \"payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz\" (UID: \"db04029d-705d-459d-b42d-03df6c5f845e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" Sep 10 19:33:15.444004 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.443983 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db04029d-705d-459d-b42d-03df6c5f845e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz\" (UID: \"db04029d-705d-459d-b42d-03df6c5f845e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" Sep 10 19:33:15.452303 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.452282 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nltzz\" (UniqueName: \"kubernetes.io/projected/db04029d-705d-459d-b42d-03df6c5f845e-kube-api-access-nltzz\") pod \"payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz\" (UID: \"db04029d-705d-459d-b42d-03df6c5f845e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" Sep 10 19:33:15.617323 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.617240 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" Sep 10 19:33:15.898051 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.898027 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz"] Sep 10 19:33:15.900562 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:33:15.900535 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb04029d_705d_459d_b42d_03df6c5f845e.slice/crio-591fdf2fb99b47a03518a48bae0112aff9c9797625dcf7270949897e5b4361d9 WatchSource:0}: Error finding container 591fdf2fb99b47a03518a48bae0112aff9c9797625dcf7270949897e5b4361d9: Status 404 returned error can't find the container with id 591fdf2fb99b47a03518a48bae0112aff9c9797625dcf7270949897e5b4361d9 Sep 10 19:33:15.984926 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.984893 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" event={"ID":"db04029d-705d-459d-b42d-03df6c5f845e","Type":"ContainerStarted","Data":"c7d15dfc99f4fd947e01a31776a5df637e91423e5eb6e8c8d154106247aadb16"} Sep 10 19:33:15.985086 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.984960 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" event={"ID":"db04029d-705d-459d-b42d-03df6c5f845e","Type":"ContainerStarted","Data":"591fdf2fb99b47a03518a48bae0112aff9c9797625dcf7270949897e5b4361d9"} Sep 10 19:33:15.985086 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:15.984989 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" Sep 10 19:33:16.015237 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:16.015188 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" podStartSLOduration=1.01516878 podStartE2EDuration="1.01516878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:33:16.014264045 +0000 UTC m=+1045.144765703" watchObservedRunningTime="2026-09-10 19:33:16.01516878 +0000 UTC m=+1045.145670414" Sep 10 19:33:26.992854 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:26.992822 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" Sep 10 19:33:58.781223 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.781187 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb"] Sep 10 19:33:58.785886 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.785869 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" Sep 10 19:33:58.788365 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.788344 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4bc8b3\"" Sep 10 19:33:58.789065 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.789050 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4bc8b3-dockercfg-4699z\"" Sep 10 19:33:58.807973 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.807955 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f48b751-f5b1-4b7c-9c9f-f3598002a8b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb\" (UID: \"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" Sep 10 19:33:58.808080 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.808007 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj46k\" (UniqueName: \"kubernetes.io/projected/2f48b751-f5b1-4b7c-9c9f-f3598002a8b7-kube-api-access-pj46k\") pod \"payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb\" (UID: \"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" Sep 10 19:33:58.809384 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.809358 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb"] Sep 10 19:33:58.853327 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.853291 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w"] Sep 10 19:33:58.857324 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.857305 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" Sep 10 19:33:58.871475 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.871448 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w"] Sep 10 19:33:58.909222 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.909197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj46k\" (UniqueName: \"kubernetes.io/projected/2f48b751-f5b1-4b7c-9c9f-f3598002a8b7-kube-api-access-pj46k\") pod \"payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb\" (UID: \"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" Sep 10 19:33:58.909344 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.909227 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbpnl\" (UniqueName: \"kubernetes.io/projected/5ca176fe-4985-4a39-97d4-d3ebaee39ee0-kube-api-access-jbpnl\") pod \"payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w\" (UID: \"5ca176fe-4985-4a39-97d4-d3ebaee39ee0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" Sep 10 19:33:58.909344 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.909253 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca176fe-4985-4a39-97d4-d3ebaee39ee0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w\" (UID: \"5ca176fe-4985-4a39-97d4-d3ebaee39ee0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" Sep 10 19:33:58.909434 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.909409 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f48b751-f5b1-4b7c-9c9f-f3598002a8b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb\" (UID: \"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" Sep 10 19:33:58.910064 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.910048 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f48b751-f5b1-4b7c-9c9f-f3598002a8b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb\" (UID: \"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" Sep 10 19:33:58.921142 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:58.921114 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj46k\" (UniqueName: \"kubernetes.io/projected/2f48b751-f5b1-4b7c-9c9f-f3598002a8b7-kube-api-access-pj46k\") pod \"payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb\" (UID: \"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" Sep 10 19:33:59.010034 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:59.010011 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbpnl\" (UniqueName: \"kubernetes.io/projected/5ca176fe-4985-4a39-97d4-d3ebaee39ee0-kube-api-access-jbpnl\") pod \"payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w\" (UID: \"5ca176fe-4985-4a39-97d4-d3ebaee39ee0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" Sep 10 19:33:59.010154 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:59.010047 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca176fe-4985-4a39-97d4-d3ebaee39ee0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w\" (UID: \"5ca176fe-4985-4a39-97d4-d3ebaee39ee0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" Sep 10 19:33:59.010683 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:59.010667 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca176fe-4985-4a39-97d4-d3ebaee39ee0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w\" (UID: \"5ca176fe-4985-4a39-97d4-d3ebaee39ee0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" Sep 10 19:33:59.018514 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:59.018496 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbpnl\" (UniqueName: \"kubernetes.io/projected/5ca176fe-4985-4a39-97d4-d3ebaee39ee0-kube-api-access-jbpnl\") pod \"payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w\" (UID: \"5ca176fe-4985-4a39-97d4-d3ebaee39ee0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" Sep 10 19:33:59.098562 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:59.098504 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" Sep 10 19:33:59.174581 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:59.174489 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" Sep 10 19:33:59.372461 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:59.372433 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb"] Sep 10 19:33:59.374903 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:33:59.374880 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f48b751_f5b1_4b7c_9c9f_f3598002a8b7.slice/crio-ec1a50124669493526e172e2f40f16f577422ff4749f93fa606402b215e40765 WatchSource:0}: Error finding container ec1a50124669493526e172e2f40f16f577422ff4749f93fa606402b215e40765: Status 404 returned error can't find the container with id ec1a50124669493526e172e2f40f16f577422ff4749f93fa606402b215e40765 Sep 10 19:33:59.453024 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:33:59.453000 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w"] Sep 10 19:33:59.455063 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:33:59.455022 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca176fe_4985_4a39_97d4_d3ebaee39ee0.slice/crio-598df124c069d13bcaa810dc690ffed359321f32f9d10fb65e6497e4b8baff4e WatchSource:0}: Error finding container 598df124c069d13bcaa810dc690ffed359321f32f9d10fb65e6497e4b8baff4e: Status 404 returned error can't find the container with id 598df124c069d13bcaa810dc690ffed359321f32f9d10fb65e6497e4b8baff4e Sep 10 19:34:00.215551 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:00.215514 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" event={"ID":"5ca176fe-4985-4a39-97d4-d3ebaee39ee0","Type":"ContainerStarted","Data":"71ba2cdc6c8ac25e7758d92ab97cc815d482747bcb5a6888371d34725fb6a14c"} Sep 10 19:34:00.216022 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:00.215596 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" event={"ID":"5ca176fe-4985-4a39-97d4-d3ebaee39ee0","Type":"ContainerStarted","Data":"598df124c069d13bcaa810dc690ffed359321f32f9d10fb65e6497e4b8baff4e"} Sep 10 19:34:00.216022 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:00.215632 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" Sep 10 19:34:00.217521 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:00.217495 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" event={"ID":"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7","Type":"ContainerStarted","Data":"5a7164a6860a5d2165cb3153ed2c8fe077091a9a03553ac10df99e44190f868f"} Sep 10 19:34:00.217635 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:00.217549 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" event={"ID":"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7","Type":"ContainerStarted","Data":"ec1a50124669493526e172e2f40f16f577422ff4749f93fa606402b215e40765"} Sep 10 19:34:00.217635 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:00.217577 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" Sep 10 19:34:00.232703 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:00.232648 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" podStartSLOduration=2.232629425 podStartE2EDuration="2.232629425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:34:00.231995855 +0000 UTC m=+1089.362497486" watchObservedRunningTime="2026-09-10 19:34:00.232629425 +0000 UTC m=+1089.363131060" Sep 10 19:34:00.249297 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:00.249254 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" podStartSLOduration=2.249239429 podStartE2EDuration="2.249239429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:34:00.24751241 +0000 UTC m=+1089.378014042" watchObservedRunningTime="2026-09-10 19:34:00.249239429 +0000 UTC m=+1089.379741093" Sep 10 19:34:11.227401 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:11.227367 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" Sep 10 19:34:11.227891 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:11.227450 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" Sep 10 19:34:38.148550 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:38.148506 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h"] Sep 10 19:34:38.152944 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:38.152928 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" Sep 10 19:34:38.156189 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:38.156161 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c615f98e\"" Sep 10 19:34:38.156189 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:38.156165 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c615f98e-dockercfg-7g9j8\"" Sep 10 19:34:38.175074 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:38.175050 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h"] Sep 10 19:34:38.242544 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:38.242516 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156a7a06-61fc-4ed5-8d69-3305e97fd383-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h\" (UID: \"156a7a06-61fc-4ed5-8d69-3305e97fd383\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" Sep 10 19:34:38.242715 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:38.242569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv666\" (UniqueName: \"kubernetes.io/projected/156a7a06-61fc-4ed5-8d69-3305e97fd383-kube-api-access-gv666\") pod \"payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h\" (UID: \"156a7a06-61fc-4ed5-8d69-3305e97fd383\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" Sep 10 19:34:38.343838 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:38.343806 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv666\" (UniqueName: \"kubernetes.io/projected/156a7a06-61fc-4ed5-8d69-3305e97fd383-kube-api-access-gv666\") pod \"payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h\" (UID: \"156a7a06-61fc-4ed5-8d69-3305e97fd383\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" Sep 10 19:34:38.343971 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:38.343917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156a7a06-61fc-4ed5-8d69-3305e97fd383-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h\" (UID: \"156a7a06-61fc-4ed5-8d69-3305e97fd383\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" Sep 10 19:34:38.344585 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:38.344566 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156a7a06-61fc-4ed5-8d69-3305e97fd383-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h\" (UID: \"156a7a06-61fc-4ed5-8d69-3305e97fd383\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" Sep 10 19:34:38.355076 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:38.355049 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv666\" (UniqueName: \"kubernetes.io/projected/156a7a06-61fc-4ed5-8d69-3305e97fd383-kube-api-access-gv666\") pod \"payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h\" (UID: \"156a7a06-61fc-4ed5-8d69-3305e97fd383\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" Sep 10 19:34:38.463365 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:38.463293 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" Sep 10 19:34:38.737146 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:38.737124 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h"] Sep 10 19:34:38.739106 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:34:38.739072 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156a7a06_61fc_4ed5_8d69_3305e97fd383.slice/crio-98791cfb50032e43427c6c5297002a7ae9111f2812ad342622eb9e68a05b4b59 WatchSource:0}: Error finding container 98791cfb50032e43427c6c5297002a7ae9111f2812ad342622eb9e68a05b4b59: Status 404 returned error can't find the container with id 98791cfb50032e43427c6c5297002a7ae9111f2812ad342622eb9e68a05b4b59 Sep 10 19:34:39.415009 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:39.414972 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" event={"ID":"156a7a06-61fc-4ed5-8d69-3305e97fd383","Type":"ContainerStarted","Data":"de0a01cf1d747297bc7d1393598d9c8098abd6c8f16126ba8781042f2873545c"} Sep 10 19:34:39.415418 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:39.415024 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" event={"ID":"156a7a06-61fc-4ed5-8d69-3305e97fd383","Type":"ContainerStarted","Data":"98791cfb50032e43427c6c5297002a7ae9111f2812ad342622eb9e68a05b4b59"} Sep 10 19:34:39.415418 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:39.415128 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" Sep 10 19:34:39.434219 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:39.434172 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" podStartSLOduration=1.434156954 podStartE2EDuration="1.434156954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:34:39.431806739 +0000 UTC m=+1128.562308371" watchObservedRunningTime="2026-09-10 19:34:39.434156954 +0000 UTC m=+1128.564658584" Sep 10 19:34:50.423543 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:50.423513 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" Sep 10 19:34:51.359069 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:51.359041 2578 scope.go:117] "RemoveContainer" containerID="3c04f7dc460485adb3295f156c081f69b197dfb238a7e578e4a1f8db6b18164e" Sep 10 19:34:57.003886 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.003856 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll"] Sep 10 19:34:57.008740 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.008705 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.011882 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.011860 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cab4ee\"/\"auth-test-model-cab4ee-kserve-self-signed-certs\"" Sep 10 19:34:57.012032 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.011892 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cab4ee\"/\"kube-root-ca.crt\"" Sep 10 19:34:57.012032 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.011961 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-cab4ee\"/\"default-dockercfg-z99gm\"" Sep 10 19:34:57.012032 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.011962 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-cab4ee\"/\"openshift-service-ca.crt\"" Sep 10 19:34:57.016393 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.016233 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll"] Sep 10 19:34:57.096654 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.096625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-tmp-dir\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.096843 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.096659 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-home\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.096843 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.096686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12ba78a-b802-465b-8554-d9c374ea19ef-tls-certs\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.096843 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.096787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-model-cache\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.096843 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.096829 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-726bp\" (UniqueName: \"kubernetes.io/projected/c12ba78a-b802-465b-8554-d9c374ea19ef-kube-api-access-726bp\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.097021 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.096930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-dshm\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.198049 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.198015 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-tmp-dir\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.198226 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.198059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-home\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.198226 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.198085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12ba78a-b802-465b-8554-d9c374ea19ef-tls-certs\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.198226 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.198116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-model-cache\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.198226 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.198140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-726bp\" (UniqueName: \"kubernetes.io/projected/c12ba78a-b802-465b-8554-d9c374ea19ef-kube-api-access-726bp\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.198226 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.198194 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-dshm\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.198493 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.198460 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-home\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.198618 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.198523 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-tmp-dir\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.198618 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.198570 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-model-cache\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.200578 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.200557 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-dshm\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.200747 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.200709 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12ba78a-b802-465b-8554-d9c374ea19ef-tls-certs\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.206782 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.206756 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-726bp\" (UniqueName: \"kubernetes.io/projected/c12ba78a-b802-465b-8554-d9c374ea19ef-kube-api-access-726bp\") pod \"auth-test-model-cab4ee-kserve-799c454cc9-rbdll\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.319608 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.319517 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:34:57.592191 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:57.592164 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll"] Sep 10 19:34:57.594083 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:34:57.594057 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc12ba78a_b802_465b_8554_d9c374ea19ef.slice/crio-47f1c21b19c5034ac67d81d44b47a059dd7938a758d2ba3017c1a3d6cc69c76c WatchSource:0}: Error finding container 47f1c21b19c5034ac67d81d44b47a059dd7938a758d2ba3017c1a3d6cc69c76c: Status 404 returned error can't find the container with id 47f1c21b19c5034ac67d81d44b47a059dd7938a758d2ba3017c1a3d6cc69c76c Sep 10 19:34:58.512633 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:58.512588 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" event={"ID":"c12ba78a-b802-465b-8554-d9c374ea19ef","Type":"ContainerStarted","Data":"b37a47366a841d0220965d8cd4d66294dcbfa2d1e75815884174bef7b6047965"} Sep 10 19:34:58.512633 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:58.512637 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" event={"ID":"c12ba78a-b802-465b-8554-d9c374ea19ef","Type":"ContainerStarted","Data":"47f1c21b19c5034ac67d81d44b47a059dd7938a758d2ba3017c1a3d6cc69c76c"} Sep 10 19:34:58.535801 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:34:58.535757 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" podStartSLOduration=2.535719814 podStartE2EDuration="2.535719814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:34:58.533631308 +0000 UTC m=+1147.664132952" watchObservedRunningTime="2026-09-10 19:34:58.535719814 +0000 UTC m=+1147.666221445" Sep 10 19:35:07.320671 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:07.320630 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:35:07.321183 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:07.320686 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:35:07.332904 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:07.332882 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:35:07.568026 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:07.567997 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:35:40.161552 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:40.161514 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm"] Sep 10 19:35:40.165110 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:40.165094 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm" Sep 10 19:35:40.167324 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:40.167304 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:35:40.167423 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:40.167303 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkgt8\"" Sep 10 19:35:40.171853 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:40.171831 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm"] Sep 10 19:35:40.286146 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:40.286111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f9081a-adfa-4d87-8eb1-a348e98313e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm\" (UID: \"f9f9081a-adfa-4d87-8eb1-a348e98313e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm" Sep 10 19:35:40.386927 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:40.386890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f9081a-adfa-4d87-8eb1-a348e98313e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm\" (UID: \"f9f9081a-adfa-4d87-8eb1-a348e98313e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm" Sep 10 19:35:40.387484 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:40.387465 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f9081a-adfa-4d87-8eb1-a348e98313e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm\" (UID: \"f9f9081a-adfa-4d87-8eb1-a348e98313e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm" Sep 10 19:35:40.476665 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:40.476600 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm" Sep 10 19:35:40.749483 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:40.749457 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm"] Sep 10 19:35:40.751036 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:35:40.751013 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f9081a_adfa_4d87_8eb1_a348e98313e7.slice/crio-9276c2c5ade163aa7da3cc5294cac73e0fd8b949df8d47042d50116a8166c5ed WatchSource:0}: Error finding container 9276c2c5ade163aa7da3cc5294cac73e0fd8b949df8d47042d50116a8166c5ed: Status 404 returned error can't find the container with id 9276c2c5ade163aa7da3cc5294cac73e0fd8b949df8d47042d50116a8166c5ed Sep 10 19:35:41.735928 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:41.735836 2578 generic.go:356] "Generic (PLEG): container finished" podID="f9f9081a-adfa-4d87-8eb1-a348e98313e7" containerID="edcd3a922dfc98ce222129cc8386a4e0d624715ba5c3665a837bbc4873b89b15" exitCode=0 Sep 10 19:35:41.736323 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:41.735925 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm" event={"ID":"f9f9081a-adfa-4d87-8eb1-a348e98313e7","Type":"ContainerDied","Data":"edcd3a922dfc98ce222129cc8386a4e0d624715ba5c3665a837bbc4873b89b15"} Sep 10 19:35:41.736323 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:41.735978 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm" event={"ID":"f9f9081a-adfa-4d87-8eb1-a348e98313e7","Type":"ContainerStarted","Data":"9276c2c5ade163aa7da3cc5294cac73e0fd8b949df8d47042d50116a8166c5ed"} Sep 10 19:35:42.891444 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:42.891421 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm" Sep 10 19:35:43.007062 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:43.006986 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f9081a-adfa-4d87-8eb1-a348e98313e7-maas-api-service-ca\") pod \"f9f9081a-adfa-4d87-8eb1-a348e98313e7\" (UID: \"f9f9081a-adfa-4d87-8eb1-a348e98313e7\") " Sep 10 19:35:43.007372 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:43.007345 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9f9081a-adfa-4d87-8eb1-a348e98313e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f9f9081a-adfa-4d87-8eb1-a348e98313e7" (UID: "f9f9081a-adfa-4d87-8eb1-a348e98313e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:35:43.107651 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:43.107619 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9f9081a-adfa-4d87-8eb1-a348e98313e7-maas-api-service-ca\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:35:43.747428 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:43.747343 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm" event={"ID":"f9f9081a-adfa-4d87-8eb1-a348e98313e7","Type":"ContainerDied","Data":"9276c2c5ade163aa7da3cc5294cac73e0fd8b949df8d47042d50116a8166c5ed"} Sep 10 19:35:43.747428 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:43.747382 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm" Sep 10 19:35:43.747428 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:43.747391 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9276c2c5ade163aa7da3cc5294cac73e0fd8b949df8d47042d50116a8166c5ed" Sep 10 19:35:51.377010 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:51.376981 2578 scope.go:117] "RemoveContainer" containerID="cd394e5b1a549ddd136ea38322f4298d7ea2ec4705b4d9f6e786cb6f4b00b550" Sep 10 19:35:51.385044 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:51.385023 2578 scope.go:117] "RemoveContainer" containerID="6fc38629665d8c159b54e391729cfffcb7a08ea660b70ae77a92f40d3266de7a" Sep 10 19:35:51.393888 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:51.393868 2578 scope.go:117] "RemoveContainer" containerID="1c3439e01dd2fb914d1173beba4f9ef7b30aa3dbe5c2e99213d00a96ca954658" Sep 10 19:35:56.003275 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.003230 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h"] Sep 10 19:35:56.003705 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.003454 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" podUID="156a7a06-61fc-4ed5-8d69-3305e97fd383" containerName="payload-pre-processing" containerID="cri-o://de0a01cf1d747297bc7d1393598d9c8098abd6c8f16126ba8781042f2873545c" gracePeriod=30 Sep 10 19:35:56.264427 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.264367 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" Sep 10 19:35:56.281129 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.281095 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm"] Sep 10 19:35:56.286143 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.286109 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c615f98e-6j2fm"] Sep 10 19:35:56.325412 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.325378 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gv666\" (UniqueName: \"kubernetes.io/projected/156a7a06-61fc-4ed5-8d69-3305e97fd383-kube-api-access-gv666\") pod \"156a7a06-61fc-4ed5-8d69-3305e97fd383\" (UID: \"156a7a06-61fc-4ed5-8d69-3305e97fd383\") " Sep 10 19:35:56.325607 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.325473 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156a7a06-61fc-4ed5-8d69-3305e97fd383-plugins-config-volume\") pod \"156a7a06-61fc-4ed5-8d69-3305e97fd383\" (UID: \"156a7a06-61fc-4ed5-8d69-3305e97fd383\") " Sep 10 19:35:56.326228 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.326102 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156a7a06-61fc-4ed5-8d69-3305e97fd383-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "156a7a06-61fc-4ed5-8d69-3305e97fd383" (UID: "156a7a06-61fc-4ed5-8d69-3305e97fd383"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:35:56.327938 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.327909 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156a7a06-61fc-4ed5-8d69-3305e97fd383-kube-api-access-gv666" (OuterVolumeSpecName: "kube-api-access-gv666") pod "156a7a06-61fc-4ed5-8d69-3305e97fd383" (UID: "156a7a06-61fc-4ed5-8d69-3305e97fd383"). InnerVolumeSpecName "kube-api-access-gv666". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:35:56.426508 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.426464 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156a7a06-61fc-4ed5-8d69-3305e97fd383-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:35:56.426508 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.426497 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv666\" (UniqueName: \"kubernetes.io/projected/156a7a06-61fc-4ed5-8d69-3305e97fd383-kube-api-access-gv666\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:35:56.815055 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.815016 2578 generic.go:356] "Generic (PLEG): container finished" podID="156a7a06-61fc-4ed5-8d69-3305e97fd383" containerID="de0a01cf1d747297bc7d1393598d9c8098abd6c8f16126ba8781042f2873545c" exitCode=0 Sep 10 19:35:56.815229 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.815066 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" Sep 10 19:35:56.815229 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.815090 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" event={"ID":"156a7a06-61fc-4ed5-8d69-3305e97fd383","Type":"ContainerDied","Data":"de0a01cf1d747297bc7d1393598d9c8098abd6c8f16126ba8781042f2873545c"} Sep 10 19:35:56.815229 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.815144 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h" event={"ID":"156a7a06-61fc-4ed5-8d69-3305e97fd383","Type":"ContainerDied","Data":"98791cfb50032e43427c6c5297002a7ae9111f2812ad342622eb9e68a05b4b59"} Sep 10 19:35:56.815229 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.815166 2578 scope.go:117] "RemoveContainer" containerID="de0a01cf1d747297bc7d1393598d9c8098abd6c8f16126ba8781042f2873545c" Sep 10 19:35:56.824743 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.824709 2578 scope.go:117] "RemoveContainer" containerID="de0a01cf1d747297bc7d1393598d9c8098abd6c8f16126ba8781042f2873545c" Sep 10 19:35:56.825020 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:35:56.825001 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de0a01cf1d747297bc7d1393598d9c8098abd6c8f16126ba8781042f2873545c\": container with ID starting with de0a01cf1d747297bc7d1393598d9c8098abd6c8f16126ba8781042f2873545c not found: ID does not exist" containerID="de0a01cf1d747297bc7d1393598d9c8098abd6c8f16126ba8781042f2873545c" Sep 10 19:35:56.825059 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.825031 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de0a01cf1d747297bc7d1393598d9c8098abd6c8f16126ba8781042f2873545c"} err="failed to get container status \"de0a01cf1d747297bc7d1393598d9c8098abd6c8f16126ba8781042f2873545c\": rpc error: code = NotFound desc = could not find container \"de0a01cf1d747297bc7d1393598d9c8098abd6c8f16126ba8781042f2873545c\": container with ID starting with de0a01cf1d747297bc7d1393598d9c8098abd6c8f16126ba8781042f2873545c not found: ID does not exist" Sep 10 19:35:56.849405 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.849373 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h"] Sep 10 19:35:56.865318 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:56.865293 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c615f98e-795f74b95d-vl86h"] Sep 10 19:35:57.392077 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:57.392039 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="156a7a06-61fc-4ed5-8d69-3305e97fd383" path="/var/lib/kubelet/pods/156a7a06-61fc-4ed5-8d69-3305e97fd383/volumes" Sep 10 19:35:57.392432 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:35:57.392360 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9f9081a-adfa-4d87-8eb1-a348e98313e7" path="/var/lib/kubelet/pods/f9f9081a-adfa-4d87-8eb1-a348e98313e7/volumes" Sep 10 19:36:51.419509 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:36:51.419441 2578 scope.go:117] "RemoveContainer" containerID="f7498ca014b8851560092b3defb1efe6edda467b43237261795cb5ed5057d859" Sep 10 19:36:51.429308 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:36:51.429285 2578 scope.go:117] "RemoveContainer" containerID="86b4e009e4735736dcb98caa74cb128f4a849219500d88d7b9f93d25ee4989f7" Sep 10 19:37:13.565813 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:13.565769 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll"] Sep 10 19:37:28.586275 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.586217 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" podUID="c12ba78a-b802-465b-8554-d9c374ea19ef" containerName="main" containerID="cri-o://b37a47366a841d0220965d8cd4d66294dcbfa2d1e75815884174bef7b6047965" gracePeriod=45 Sep 10 19:37:28.839175 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.839121 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:37:28.924971 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.924938 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-tmp-dir\") pod \"c12ba78a-b802-465b-8554-d9c374ea19ef\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " Sep 10 19:37:28.925165 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.925005 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12ba78a-b802-465b-8554-d9c374ea19ef-tls-certs\") pod \"c12ba78a-b802-465b-8554-d9c374ea19ef\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " Sep 10 19:37:28.925165 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.925021 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-dshm\") pod \"c12ba78a-b802-465b-8554-d9c374ea19ef\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " Sep 10 19:37:28.925165 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.925048 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-home\") pod \"c12ba78a-b802-465b-8554-d9c374ea19ef\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " Sep 10 19:37:28.925165 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.925082 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-726bp\" (UniqueName: \"kubernetes.io/projected/c12ba78a-b802-465b-8554-d9c374ea19ef-kube-api-access-726bp\") pod \"c12ba78a-b802-465b-8554-d9c374ea19ef\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " Sep 10 19:37:28.925165 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.925141 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-model-cache\") pod \"c12ba78a-b802-465b-8554-d9c374ea19ef\" (UID: \"c12ba78a-b802-465b-8554-d9c374ea19ef\") " Sep 10 19:37:28.925432 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.925225 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c12ba78a-b802-465b-8554-d9c374ea19ef" (UID: "c12ba78a-b802-465b-8554-d9c374ea19ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:28.925432 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.925366 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-home" (OuterVolumeSpecName: "home") pod "c12ba78a-b802-465b-8554-d9c374ea19ef" (UID: "c12ba78a-b802-465b-8554-d9c374ea19ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:28.925432 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.925420 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-tmp-dir\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:37:28.925547 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.925446 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-model-cache" (OuterVolumeSpecName: "model-cache") pod "c12ba78a-b802-465b-8554-d9c374ea19ef" (UID: "c12ba78a-b802-465b-8554-d9c374ea19ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:28.927544 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.927514 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12ba78a-b802-465b-8554-d9c374ea19ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c12ba78a-b802-465b-8554-d9c374ea19ef" (UID: "c12ba78a-b802-465b-8554-d9c374ea19ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 19:37:28.927544 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.927520 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12ba78a-b802-465b-8554-d9c374ea19ef-kube-api-access-726bp" (OuterVolumeSpecName: "kube-api-access-726bp") pod "c12ba78a-b802-465b-8554-d9c374ea19ef" (UID: "c12ba78a-b802-465b-8554-d9c374ea19ef"). InnerVolumeSpecName "kube-api-access-726bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:37:28.927544 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:28.927517 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-dshm" (OuterVolumeSpecName: "dshm") pod "c12ba78a-b802-465b-8554-d9c374ea19ef" (UID: "c12ba78a-b802-465b-8554-d9c374ea19ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:37:29.026128 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.026088 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12ba78a-b802-465b-8554-d9c374ea19ef-tls-certs\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:37:29.026128 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.026120 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-dshm\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:37:29.026128 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.026129 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-home\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:37:29.026128 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.026137 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-726bp\" (UniqueName: \"kubernetes.io/projected/c12ba78a-b802-465b-8554-d9c374ea19ef-kube-api-access-726bp\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:37:29.026388 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.026145 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12ba78a-b802-465b-8554-d9c374ea19ef-model-cache\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:37:29.275925 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.275888 2578 generic.go:356] "Generic (PLEG): container finished" podID="c12ba78a-b802-465b-8554-d9c374ea19ef" containerID="b37a47366a841d0220965d8cd4d66294dcbfa2d1e75815884174bef7b6047965" exitCode=0 Sep 10 19:37:29.276082 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.275940 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" Sep 10 19:37:29.276082 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.275958 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" event={"ID":"c12ba78a-b802-465b-8554-d9c374ea19ef","Type":"ContainerDied","Data":"b37a47366a841d0220965d8cd4d66294dcbfa2d1e75815884174bef7b6047965"} Sep 10 19:37:29.276082 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.275998 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll" event={"ID":"c12ba78a-b802-465b-8554-d9c374ea19ef","Type":"ContainerDied","Data":"47f1c21b19c5034ac67d81d44b47a059dd7938a758d2ba3017c1a3d6cc69c76c"} Sep 10 19:37:29.276082 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.276020 2578 scope.go:117] "RemoveContainer" containerID="b37a47366a841d0220965d8cd4d66294dcbfa2d1e75815884174bef7b6047965" Sep 10 19:37:29.285164 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.285127 2578 scope.go:117] "RemoveContainer" containerID="b37a47366a841d0220965d8cd4d66294dcbfa2d1e75815884174bef7b6047965" Sep 10 19:37:29.285407 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:37:29.285391 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37a47366a841d0220965d8cd4d66294dcbfa2d1e75815884174bef7b6047965\": container with ID starting with b37a47366a841d0220965d8cd4d66294dcbfa2d1e75815884174bef7b6047965 not found: ID does not exist" containerID="b37a47366a841d0220965d8cd4d66294dcbfa2d1e75815884174bef7b6047965" Sep 10 19:37:29.285455 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.285415 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37a47366a841d0220965d8cd4d66294dcbfa2d1e75815884174bef7b6047965"} err="failed to get container status \"b37a47366a841d0220965d8cd4d66294dcbfa2d1e75815884174bef7b6047965\": rpc error: code = NotFound desc = could not find container \"b37a47366a841d0220965d8cd4d66294dcbfa2d1e75815884174bef7b6047965\": container with ID starting with b37a47366a841d0220965d8cd4d66294dcbfa2d1e75815884174bef7b6047965 not found: ID does not exist" Sep 10 19:37:29.296591 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.296563 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll"] Sep 10 19:37:29.300703 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.300679 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-cab4ee/auth-test-model-cab4ee-kserve-799c454cc9-rbdll"] Sep 10 19:37:29.392488 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:29.392460 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c12ba78a-b802-465b-8554-d9c374ea19ef" path="/var/lib/kubelet/pods/c12ba78a-b802-465b-8554-d9c374ea19ef/volumes" Sep 10 19:37:51.454967 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:51.454938 2578 scope.go:117] "RemoveContainer" containerID="ab39d4dfd85a96ca153d0ea19286cf0b074d78e596835ad2d0fffc965b495e6d" Sep 10 19:37:54.286980 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.286949 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn"] Sep 10 19:37:54.287490 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.287472 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12ba78a-b802-465b-8554-d9c374ea19ef" containerName="main" Sep 10 19:37:54.287577 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.287493 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12ba78a-b802-465b-8554-d9c374ea19ef" containerName="main" Sep 10 19:37:54.287577 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.287517 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156a7a06-61fc-4ed5-8d69-3305e97fd383" containerName="payload-pre-processing" Sep 10 19:37:54.287577 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.287525 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="156a7a06-61fc-4ed5-8d69-3305e97fd383" containerName="payload-pre-processing" Sep 10 19:37:54.287577 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.287533 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9f9081a-adfa-4d87-8eb1-a348e98313e7" containerName="revoke-keys" Sep 10 19:37:54.287577 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.287542 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9f9081a-adfa-4d87-8eb1-a348e98313e7" containerName="revoke-keys" Sep 10 19:37:54.287755 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.287619 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="156a7a06-61fc-4ed5-8d69-3305e97fd383" containerName="payload-pre-processing" Sep 10 19:37:54.287755 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.287636 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c12ba78a-b802-465b-8554-d9c374ea19ef" containerName="main" Sep 10 19:37:54.287755 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.287647 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9f9081a-adfa-4d87-8eb1-a348e98313e7" containerName="revoke-keys" Sep 10 19:37:54.292212 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.292192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" Sep 10 19:37:54.294681 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.294652 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-439275\"" Sep 10 19:37:54.296827 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.296808 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn"] Sep 10 19:37:54.300664 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.300645 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" Sep 10 19:37:54.300919 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.300897 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn"] Sep 10 19:37:54.302664 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.302642 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-439275-dockercfg-ncqhz\"" Sep 10 19:37:54.314133 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.314108 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn"] Sep 10 19:37:54.343426 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.343396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2846dac5-1949-4a75-a642-78afa838ef89-plugins-config-volume\") pod \"payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn\" (UID: \"2846dac5-1949-4a75-a642-78afa838ef89\") " pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" Sep 10 19:37:54.343549 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.343439 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b153b56-af63-4517-bbd6-9a53b226146d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn\" (UID: \"6b153b56-af63-4517-bbd6-9a53b226146d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" Sep 10 19:37:54.343549 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.343509 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v49g2\" (UniqueName: \"kubernetes.io/projected/6b153b56-af63-4517-bbd6-9a53b226146d-kube-api-access-v49g2\") pod \"payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn\" (UID: \"6b153b56-af63-4517-bbd6-9a53b226146d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" Sep 10 19:37:54.343626 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.343589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrgdl\" (UniqueName: \"kubernetes.io/projected/2846dac5-1949-4a75-a642-78afa838ef89-kube-api-access-rrgdl\") pod \"payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn\" (UID: \"2846dac5-1949-4a75-a642-78afa838ef89\") " pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" Sep 10 19:37:54.444901 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.444868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v49g2\" (UniqueName: \"kubernetes.io/projected/6b153b56-af63-4517-bbd6-9a53b226146d-kube-api-access-v49g2\") pod \"payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn\" (UID: \"6b153b56-af63-4517-bbd6-9a53b226146d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" Sep 10 19:37:54.445072 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.444949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrgdl\" (UniqueName: \"kubernetes.io/projected/2846dac5-1949-4a75-a642-78afa838ef89-kube-api-access-rrgdl\") pod \"payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn\" (UID: \"2846dac5-1949-4a75-a642-78afa838ef89\") " pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" Sep 10 19:37:54.445072 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.444975 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2846dac5-1949-4a75-a642-78afa838ef89-plugins-config-volume\") pod \"payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn\" (UID: \"2846dac5-1949-4a75-a642-78afa838ef89\") " pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" Sep 10 19:37:54.445072 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.445014 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b153b56-af63-4517-bbd6-9a53b226146d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn\" (UID: \"6b153b56-af63-4517-bbd6-9a53b226146d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" Sep 10 19:37:54.445653 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.445630 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b153b56-af63-4517-bbd6-9a53b226146d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn\" (UID: \"6b153b56-af63-4517-bbd6-9a53b226146d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" Sep 10 19:37:54.445771 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.445658 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2846dac5-1949-4a75-a642-78afa838ef89-plugins-config-volume\") pod \"payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn\" (UID: \"2846dac5-1949-4a75-a642-78afa838ef89\") " pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" Sep 10 19:37:54.453587 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.453562 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrgdl\" (UniqueName: \"kubernetes.io/projected/2846dac5-1949-4a75-a642-78afa838ef89-kube-api-access-rrgdl\") pod \"payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn\" (UID: \"2846dac5-1949-4a75-a642-78afa838ef89\") " pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" Sep 10 19:37:54.453678 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.453663 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v49g2\" (UniqueName: \"kubernetes.io/projected/6b153b56-af63-4517-bbd6-9a53b226146d-kube-api-access-v49g2\") pod \"payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn\" (UID: \"6b153b56-af63-4517-bbd6-9a53b226146d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" Sep 10 19:37:54.603569 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.603495 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" Sep 10 19:37:54.617628 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.617608 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" Sep 10 19:37:54.874818 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.874792 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn"] Sep 10 19:37:54.876425 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:37:54.876399 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b153b56_af63_4517_bbd6_9a53b226146d.slice/crio-a04669e5c0df30b822ee7ffe2d8430b2920af00fea014c8d368303b8cfa4a0c1 WatchSource:0}: Error finding container a04669e5c0df30b822ee7ffe2d8430b2920af00fea014c8d368303b8cfa4a0c1: Status 404 returned error can't find the container with id a04669e5c0df30b822ee7ffe2d8430b2920af00fea014c8d368303b8cfa4a0c1 Sep 10 19:37:54.896288 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:54.896266 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn"] Sep 10 19:37:54.897559 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:37:54.897531 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2846dac5_1949_4a75_a642_78afa838ef89.slice/crio-d618f02e37802575af18a8de71c26b3290db6296c522efedcb870ff552920801 WatchSource:0}: Error finding container d618f02e37802575af18a8de71c26b3290db6296c522efedcb870ff552920801: Status 404 returned error can't find the container with id d618f02e37802575af18a8de71c26b3290db6296c522efedcb870ff552920801 Sep 10 19:37:55.409760 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:55.409699 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" event={"ID":"6b153b56-af63-4517-bbd6-9a53b226146d","Type":"ContainerStarted","Data":"0d9b2a2aba2d89517f67294899f2d556265118ca46c604a7749d0192aed7018f"} Sep 10 19:37:55.410198 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:55.409785 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" Sep 10 19:37:55.410198 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:55.409810 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" event={"ID":"6b153b56-af63-4517-bbd6-9a53b226146d","Type":"ContainerStarted","Data":"a04669e5c0df30b822ee7ffe2d8430b2920af00fea014c8d368303b8cfa4a0c1"} Sep 10 19:37:55.411472 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:55.411443 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" event={"ID":"2846dac5-1949-4a75-a642-78afa838ef89","Type":"ContainerStarted","Data":"ed6393cd2f60f8ba37e240fe8ed846a2d11070e33a3aaefc59de245ff3fac83f"} Sep 10 19:37:55.411584 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:55.411490 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" event={"ID":"2846dac5-1949-4a75-a642-78afa838ef89","Type":"ContainerStarted","Data":"d618f02e37802575af18a8de71c26b3290db6296c522efedcb870ff552920801"} Sep 10 19:37:55.411584 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:55.411529 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" Sep 10 19:37:55.429529 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:55.429485 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" podStartSLOduration=1.4294728239999999 podStartE2EDuration="1.429472824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:37:55.4272224 +0000 UTC m=+1324.557724038" watchObservedRunningTime="2026-09-10 19:37:55.429472824 +0000 UTC m=+1324.559974454" Sep 10 19:37:55.444266 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:37:55.444227 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" podStartSLOduration=1.44421547 podStartE2EDuration="1.44421547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:37:55.442904741 +0000 UTC m=+1324.573406373" watchObservedRunningTime="2026-09-10 19:37:55.44421547 +0000 UTC m=+1324.574717102" Sep 10 19:38:06.418663 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:06.418634 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" Sep 10 19:38:06.419045 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:06.418695 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" Sep 10 19:38:23.013821 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.013784 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl"] Sep 10 19:38:23.017218 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.017197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" Sep 10 19:38:23.019514 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.019495 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cbac1f-dockercfg-qvwxc\"" Sep 10 19:38:23.019900 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.019881 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cbac1f\"" Sep 10 19:38:23.029827 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.029799 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl"] Sep 10 19:38:23.073483 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.073456 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c"] Sep 10 19:38:23.076870 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.076856 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" Sep 10 19:38:23.087184 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.087165 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c"] Sep 10 19:38:23.093072 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.093049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2kb2\" (UniqueName: \"kubernetes.io/projected/f200e919-64e2-4432-95f2-017382cf700e-kube-api-access-k2kb2\") pod \"payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c\" (UID: \"f200e919-64e2-4432-95f2-017382cf700e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" Sep 10 19:38:23.093165 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.093085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f200e919-64e2-4432-95f2-017382cf700e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c\" (UID: \"f200e919-64e2-4432-95f2-017382cf700e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" Sep 10 19:38:23.093165 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.093118 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdb8698-93d6-47f2-849f-ed49a69e450d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl\" (UID: \"8cdb8698-93d6-47f2-849f-ed49a69e450d\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" Sep 10 19:38:23.093165 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.093161 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv6v4\" (UniqueName: \"kubernetes.io/projected/8cdb8698-93d6-47f2-849f-ed49a69e450d-kube-api-access-rv6v4\") pod \"payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl\" (UID: \"8cdb8698-93d6-47f2-849f-ed49a69e450d\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" Sep 10 19:38:23.193694 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.193650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kb2\" (UniqueName: \"kubernetes.io/projected/f200e919-64e2-4432-95f2-017382cf700e-kube-api-access-k2kb2\") pod \"payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c\" (UID: \"f200e919-64e2-4432-95f2-017382cf700e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" Sep 10 19:38:23.193694 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.193691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f200e919-64e2-4432-95f2-017382cf700e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c\" (UID: \"f200e919-64e2-4432-95f2-017382cf700e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" Sep 10 19:38:23.193959 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.193740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdb8698-93d6-47f2-849f-ed49a69e450d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl\" (UID: \"8cdb8698-93d6-47f2-849f-ed49a69e450d\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" Sep 10 19:38:23.193959 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.193789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv6v4\" (UniqueName: \"kubernetes.io/projected/8cdb8698-93d6-47f2-849f-ed49a69e450d-kube-api-access-rv6v4\") pod \"payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl\" (UID: \"8cdb8698-93d6-47f2-849f-ed49a69e450d\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" Sep 10 19:38:23.194420 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.194396 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdb8698-93d6-47f2-849f-ed49a69e450d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl\" (UID: \"8cdb8698-93d6-47f2-849f-ed49a69e450d\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" Sep 10 19:38:23.194528 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.194398 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f200e919-64e2-4432-95f2-017382cf700e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c\" (UID: \"f200e919-64e2-4432-95f2-017382cf700e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" Sep 10 19:38:23.204289 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.204260 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv6v4\" (UniqueName: \"kubernetes.io/projected/8cdb8698-93d6-47f2-849f-ed49a69e450d-kube-api-access-rv6v4\") pod \"payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl\" (UID: \"8cdb8698-93d6-47f2-849f-ed49a69e450d\") " pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" Sep 10 19:38:23.204556 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.204538 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kb2\" (UniqueName: \"kubernetes.io/projected/f200e919-64e2-4432-95f2-017382cf700e-kube-api-access-k2kb2\") pod \"payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c\" (UID: \"f200e919-64e2-4432-95f2-017382cf700e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" Sep 10 19:38:23.328320 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.328255 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" Sep 10 19:38:23.387861 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.387145 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" Sep 10 19:38:23.598750 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.598709 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl"] Sep 10 19:38:23.600874 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:38:23.600843 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdb8698_93d6_47f2_849f_ed49a69e450d.slice/crio-fa23091923c158323cb70a6838215eb4772fd734b35f308bd8e8d97a169a9686 WatchSource:0}: Error finding container fa23091923c158323cb70a6838215eb4772fd734b35f308bd8e8d97a169a9686: Status 404 returned error can't find the container with id fa23091923c158323cb70a6838215eb4772fd734b35f308bd8e8d97a169a9686 Sep 10 19:38:23.664401 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:23.664372 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c"] Sep 10 19:38:23.665474 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:38:23.665451 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf200e919_64e2_4432_95f2_017382cf700e.slice/crio-af9b9010b7760bf4fbdcf4f1cf8217021d8ac4561f4c05c2bc273d36afb1243d WatchSource:0}: Error finding container af9b9010b7760bf4fbdcf4f1cf8217021d8ac4561f4c05c2bc273d36afb1243d: Status 404 returned error can't find the container with id af9b9010b7760bf4fbdcf4f1cf8217021d8ac4561f4c05c2bc273d36afb1243d Sep 10 19:38:24.565791 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:24.565748 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" event={"ID":"8cdb8698-93d6-47f2-849f-ed49a69e450d","Type":"ContainerStarted","Data":"c87c05ce8ebd9eb857810c86c83111d20872a05e3649741cce1bdc60b5d17ac5"} Sep 10 19:38:24.566334 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:24.565817 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" event={"ID":"8cdb8698-93d6-47f2-849f-ed49a69e450d","Type":"ContainerStarted","Data":"fa23091923c158323cb70a6838215eb4772fd734b35f308bd8e8d97a169a9686"} Sep 10 19:38:24.566334 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:24.565844 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" Sep 10 19:38:24.568376 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:24.568334 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" event={"ID":"f200e919-64e2-4432-95f2-017382cf700e","Type":"ContainerStarted","Data":"20de745812a33743392d57e3225095a4a63bd553ba204b1a7f1260a3b1f6d4a6"} Sep 10 19:38:24.568503 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:24.568391 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" event={"ID":"f200e919-64e2-4432-95f2-017382cf700e","Type":"ContainerStarted","Data":"af9b9010b7760bf4fbdcf4f1cf8217021d8ac4561f4c05c2bc273d36afb1243d"} Sep 10 19:38:24.568503 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:24.568490 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" Sep 10 19:38:24.582959 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:24.582917 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" podStartSLOduration=2.582903195 podStartE2EDuration="2.582903195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:38:24.581000713 +0000 UTC m=+1353.711502345" watchObservedRunningTime="2026-09-10 19:38:24.582903195 +0000 UTC m=+1353.713404826" Sep 10 19:38:24.595869 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:24.595824 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" podStartSLOduration=1.595804898 podStartE2EDuration="1.595804898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:38:24.595477391 +0000 UTC m=+1353.725979023" watchObservedRunningTime="2026-09-10 19:38:24.595804898 +0000 UTC m=+1353.726306529" Sep 10 19:38:35.575742 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:35.575706 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" Sep 10 19:38:35.576162 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:35.575891 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" Sep 10 19:38:51.476096 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:51.476067 2578 scope.go:117] "RemoveContainer" containerID="abfb23b00f0bf8cb691bc2329c4e45fbbc1e710cf03b509db958bf7bbc401f2a" Sep 10 19:38:51.484020 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:38:51.484002 2578 scope.go:117] "RemoveContainer" containerID="9349504668c2eea8d5f2a7235579d0fc6a5092fa9cc3cb2620a255d56dabadc3" Sep 10 19:39:04.935660 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:04.935625 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn"] Sep 10 19:39:04.936193 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:04.935959 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" podUID="2846dac5-1949-4a75-a642-78afa838ef89" containerName="payload-processing" containerID="cri-o://ed6393cd2f60f8ba37e240fe8ed846a2d11070e33a3aaefc59de245ff3fac83f" gracePeriod=30 Sep 10 19:39:04.942860 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:04.942714 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn"] Sep 10 19:39:04.942995 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:04.942952 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" podUID="6b153b56-af63-4517-bbd6-9a53b226146d" containerName="payload-pre-processing" containerID="cri-o://0d9b2a2aba2d89517f67294899f2d556265118ca46c604a7749d0192aed7018f" gracePeriod=30 Sep 10 19:39:05.220709 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.220685 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" Sep 10 19:39:05.223881 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.223859 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" Sep 10 19:39:05.365040 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.365011 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrgdl\" (UniqueName: \"kubernetes.io/projected/2846dac5-1949-4a75-a642-78afa838ef89-kube-api-access-rrgdl\") pod \"2846dac5-1949-4a75-a642-78afa838ef89\" (UID: \"2846dac5-1949-4a75-a642-78afa838ef89\") " Sep 10 19:39:05.365231 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.365058 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v49g2\" (UniqueName: \"kubernetes.io/projected/6b153b56-af63-4517-bbd6-9a53b226146d-kube-api-access-v49g2\") pod \"6b153b56-af63-4517-bbd6-9a53b226146d\" (UID: \"6b153b56-af63-4517-bbd6-9a53b226146d\") " Sep 10 19:39:05.365231 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.365106 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b153b56-af63-4517-bbd6-9a53b226146d-plugins-config-volume\") pod \"6b153b56-af63-4517-bbd6-9a53b226146d\" (UID: \"6b153b56-af63-4517-bbd6-9a53b226146d\") " Sep 10 19:39:05.365231 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.365143 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2846dac5-1949-4a75-a642-78afa838ef89-plugins-config-volume\") pod \"2846dac5-1949-4a75-a642-78afa838ef89\" (UID: \"2846dac5-1949-4a75-a642-78afa838ef89\") " Sep 10 19:39:05.365529 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.365501 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b153b56-af63-4517-bbd6-9a53b226146d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b153b56-af63-4517-bbd6-9a53b226146d" (UID: "6b153b56-af63-4517-bbd6-9a53b226146d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:39:05.365596 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.365522 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2846dac5-1949-4a75-a642-78afa838ef89-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2846dac5-1949-4a75-a642-78afa838ef89" (UID: "2846dac5-1949-4a75-a642-78afa838ef89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:39:05.367292 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.367262 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b153b56-af63-4517-bbd6-9a53b226146d-kube-api-access-v49g2" (OuterVolumeSpecName: "kube-api-access-v49g2") pod "6b153b56-af63-4517-bbd6-9a53b226146d" (UID: "6b153b56-af63-4517-bbd6-9a53b226146d"). InnerVolumeSpecName "kube-api-access-v49g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:39:05.367382 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.367338 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2846dac5-1949-4a75-a642-78afa838ef89-kube-api-access-rrgdl" (OuterVolumeSpecName: "kube-api-access-rrgdl") pod "2846dac5-1949-4a75-a642-78afa838ef89" (UID: "2846dac5-1949-4a75-a642-78afa838ef89"). InnerVolumeSpecName "kube-api-access-rrgdl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:39:05.466058 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.465965 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v49g2\" (UniqueName: \"kubernetes.io/projected/6b153b56-af63-4517-bbd6-9a53b226146d-kube-api-access-v49g2\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:39:05.466058 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.465998 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b153b56-af63-4517-bbd6-9a53b226146d-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:39:05.466058 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.466009 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2846dac5-1949-4a75-a642-78afa838ef89-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:39:05.466058 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.466017 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrgdl\" (UniqueName: \"kubernetes.io/projected/2846dac5-1949-4a75-a642-78afa838ef89-kube-api-access-rrgdl\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:39:05.774806 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.774699 2578 generic.go:356] "Generic (PLEG): container finished" podID="6b153b56-af63-4517-bbd6-9a53b226146d" containerID="0d9b2a2aba2d89517f67294899f2d556265118ca46c604a7749d0192aed7018f" exitCode=0 Sep 10 19:39:05.774806 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.774771 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" event={"ID":"6b153b56-af63-4517-bbd6-9a53b226146d","Type":"ContainerDied","Data":"0d9b2a2aba2d89517f67294899f2d556265118ca46c604a7749d0192aed7018f"} Sep 10 19:39:05.775062 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.774781 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" Sep 10 19:39:05.775062 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.774860 2578 scope.go:117] "RemoveContainer" containerID="0d9b2a2aba2d89517f67294899f2d556265118ca46c604a7749d0192aed7018f" Sep 10 19:39:05.775062 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.774845 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn" event={"ID":"6b153b56-af63-4517-bbd6-9a53b226146d","Type":"ContainerDied","Data":"a04669e5c0df30b822ee7ffe2d8430b2920af00fea014c8d368303b8cfa4a0c1"} Sep 10 19:39:05.776258 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.776238 2578 generic.go:356] "Generic (PLEG): container finished" podID="2846dac5-1949-4a75-a642-78afa838ef89" containerID="ed6393cd2f60f8ba37e240fe8ed846a2d11070e33a3aaefc59de245ff3fac83f" exitCode=0 Sep 10 19:39:05.776338 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.776268 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" Sep 10 19:39:05.776338 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.776323 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" event={"ID":"2846dac5-1949-4a75-a642-78afa838ef89","Type":"ContainerDied","Data":"ed6393cd2f60f8ba37e240fe8ed846a2d11070e33a3aaefc59de245ff3fac83f"} Sep 10 19:39:05.776412 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.776371 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn" event={"ID":"2846dac5-1949-4a75-a642-78afa838ef89","Type":"ContainerDied","Data":"d618f02e37802575af18a8de71c26b3290db6296c522efedcb870ff552920801"} Sep 10 19:39:05.783786 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.783768 2578 scope.go:117] "RemoveContainer" containerID="0d9b2a2aba2d89517f67294899f2d556265118ca46c604a7749d0192aed7018f" Sep 10 19:39:05.784129 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:39:05.784107 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9b2a2aba2d89517f67294899f2d556265118ca46c604a7749d0192aed7018f\": container with ID starting with 0d9b2a2aba2d89517f67294899f2d556265118ca46c604a7749d0192aed7018f not found: ID does not exist" containerID="0d9b2a2aba2d89517f67294899f2d556265118ca46c604a7749d0192aed7018f" Sep 10 19:39:05.784194 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.784154 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9b2a2aba2d89517f67294899f2d556265118ca46c604a7749d0192aed7018f"} err="failed to get container status \"0d9b2a2aba2d89517f67294899f2d556265118ca46c604a7749d0192aed7018f\": rpc error: code = NotFound desc = could not find container \"0d9b2a2aba2d89517f67294899f2d556265118ca46c604a7749d0192aed7018f\": container with ID starting with 0d9b2a2aba2d89517f67294899f2d556265118ca46c604a7749d0192aed7018f not found: ID does not exist" Sep 10 19:39:05.784194 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.784176 2578 scope.go:117] "RemoveContainer" containerID="ed6393cd2f60f8ba37e240fe8ed846a2d11070e33a3aaefc59de245ff3fac83f" Sep 10 19:39:05.793835 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.793810 2578 scope.go:117] "RemoveContainer" containerID="ed6393cd2f60f8ba37e240fe8ed846a2d11070e33a3aaefc59de245ff3fac83f" Sep 10 19:39:05.794186 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:39:05.794166 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6393cd2f60f8ba37e240fe8ed846a2d11070e33a3aaefc59de245ff3fac83f\": container with ID starting with ed6393cd2f60f8ba37e240fe8ed846a2d11070e33a3aaefc59de245ff3fac83f not found: ID does not exist" containerID="ed6393cd2f60f8ba37e240fe8ed846a2d11070e33a3aaefc59de245ff3fac83f" Sep 10 19:39:05.794263 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.794193 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6393cd2f60f8ba37e240fe8ed846a2d11070e33a3aaefc59de245ff3fac83f"} err="failed to get container status \"ed6393cd2f60f8ba37e240fe8ed846a2d11070e33a3aaefc59de245ff3fac83f\": rpc error: code = NotFound desc = could not find container \"ed6393cd2f60f8ba37e240fe8ed846a2d11070e33a3aaefc59de245ff3fac83f\": container with ID starting with ed6393cd2f60f8ba37e240fe8ed846a2d11070e33a3aaefc59de245ff3fac83f not found: ID does not exist" Sep 10 19:39:05.794328 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.794285 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn"] Sep 10 19:39:05.797805 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.797780 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-439275-6d855c68cd-vw4pn"] Sep 10 19:39:05.808719 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.808698 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn"] Sep 10 19:39:05.812241 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:05.812221 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-439275-5b68bd7bc9-8rrjn"] Sep 10 19:39:07.391632 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:07.391597 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2846dac5-1949-4a75-a642-78afa838ef89" path="/var/lib/kubelet/pods/2846dac5-1949-4a75-a642-78afa838ef89/volumes" Sep 10 19:39:07.392014 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:07.391958 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b153b56-af63-4517-bbd6-9a53b226146d" path="/var/lib/kubelet/pods/6b153b56-af63-4517-bbd6-9a53b226146d/volumes" Sep 10 19:39:27.952142 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:27.952102 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl"] Sep 10 19:39:27.952606 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:27.952327 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" podUID="8cdb8698-93d6-47f2-849f-ed49a69e450d" containerName="payload-processing" containerID="cri-o://c87c05ce8ebd9eb857810c86c83111d20872a05e3649741cce1bdc60b5d17ac5" gracePeriod=30 Sep 10 19:39:27.956874 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:27.956847 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c"] Sep 10 19:39:27.957286 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:27.957233 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" podUID="f200e919-64e2-4432-95f2-017382cf700e" containerName="payload-pre-processing" containerID="cri-o://20de745812a33743392d57e3225095a4a63bd553ba204b1a7f1260a3b1f6d4a6" gracePeriod=30 Sep 10 19:39:28.230794 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.229181 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" Sep 10 19:39:28.237278 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.237244 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" Sep 10 19:39:28.263621 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.263590 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f200e919-64e2-4432-95f2-017382cf700e-plugins-config-volume\") pod \"f200e919-64e2-4432-95f2-017382cf700e\" (UID: \"f200e919-64e2-4432-95f2-017382cf700e\") " Sep 10 19:39:28.263849 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.263710 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rv6v4\" (UniqueName: \"kubernetes.io/projected/8cdb8698-93d6-47f2-849f-ed49a69e450d-kube-api-access-rv6v4\") pod \"8cdb8698-93d6-47f2-849f-ed49a69e450d\" (UID: \"8cdb8698-93d6-47f2-849f-ed49a69e450d\") " Sep 10 19:39:28.263849 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.263820 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2kb2\" (UniqueName: \"kubernetes.io/projected/f200e919-64e2-4432-95f2-017382cf700e-kube-api-access-k2kb2\") pod \"f200e919-64e2-4432-95f2-017382cf700e\" (UID: \"f200e919-64e2-4432-95f2-017382cf700e\") " Sep 10 19:39:28.263993 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.263874 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdb8698-93d6-47f2-849f-ed49a69e450d-plugins-config-volume\") pod \"8cdb8698-93d6-47f2-849f-ed49a69e450d\" (UID: \"8cdb8698-93d6-47f2-849f-ed49a69e450d\") " Sep 10 19:39:28.265655 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.264436 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f200e919-64e2-4432-95f2-017382cf700e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f200e919-64e2-4432-95f2-017382cf700e" (UID: "f200e919-64e2-4432-95f2-017382cf700e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:39:28.265655 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.264835 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cdb8698-93d6-47f2-849f-ed49a69e450d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8cdb8698-93d6-47f2-849f-ed49a69e450d" (UID: "8cdb8698-93d6-47f2-849f-ed49a69e450d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:39:28.265655 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.265117 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cdb8698-93d6-47f2-849f-ed49a69e450d-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:39:28.265655 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.265160 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f200e919-64e2-4432-95f2-017382cf700e-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:39:28.266955 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.266925 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cdb8698-93d6-47f2-849f-ed49a69e450d-kube-api-access-rv6v4" (OuterVolumeSpecName: "kube-api-access-rv6v4") pod "8cdb8698-93d6-47f2-849f-ed49a69e450d" (UID: "8cdb8698-93d6-47f2-849f-ed49a69e450d"). InnerVolumeSpecName "kube-api-access-rv6v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:39:28.273144 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.273092 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f200e919-64e2-4432-95f2-017382cf700e-kube-api-access-k2kb2" (OuterVolumeSpecName: "kube-api-access-k2kb2") pod "f200e919-64e2-4432-95f2-017382cf700e" (UID: "f200e919-64e2-4432-95f2-017382cf700e"). InnerVolumeSpecName "kube-api-access-k2kb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:39:28.366018 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.365992 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv6v4\" (UniqueName: \"kubernetes.io/projected/8cdb8698-93d6-47f2-849f-ed49a69e450d-kube-api-access-rv6v4\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:39:28.366018 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.366019 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2kb2\" (UniqueName: \"kubernetes.io/projected/f200e919-64e2-4432-95f2-017382cf700e-kube-api-access-k2kb2\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:39:28.890316 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.890274 2578 generic.go:356] "Generic (PLEG): container finished" podID="8cdb8698-93d6-47f2-849f-ed49a69e450d" containerID="c87c05ce8ebd9eb857810c86c83111d20872a05e3649741cce1bdc60b5d17ac5" exitCode=0 Sep 10 19:39:28.890316 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.890315 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" Sep 10 19:39:28.890573 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.890350 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" event={"ID":"8cdb8698-93d6-47f2-849f-ed49a69e450d","Type":"ContainerDied","Data":"c87c05ce8ebd9eb857810c86c83111d20872a05e3649741cce1bdc60b5d17ac5"} Sep 10 19:39:28.890573 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.890405 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl" event={"ID":"8cdb8698-93d6-47f2-849f-ed49a69e450d","Type":"ContainerDied","Data":"fa23091923c158323cb70a6838215eb4772fd734b35f308bd8e8d97a169a9686"} Sep 10 19:39:28.890573 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.890433 2578 scope.go:117] "RemoveContainer" containerID="c87c05ce8ebd9eb857810c86c83111d20872a05e3649741cce1bdc60b5d17ac5" Sep 10 19:39:28.891957 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.891933 2578 generic.go:356] "Generic (PLEG): container finished" podID="f200e919-64e2-4432-95f2-017382cf700e" containerID="20de745812a33743392d57e3225095a4a63bd553ba204b1a7f1260a3b1f6d4a6" exitCode=0 Sep 10 19:39:28.892101 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.891980 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" Sep 10 19:39:28.892101 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.891968 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" event={"ID":"f200e919-64e2-4432-95f2-017382cf700e","Type":"ContainerDied","Data":"20de745812a33743392d57e3225095a4a63bd553ba204b1a7f1260a3b1f6d4a6"} Sep 10 19:39:28.892220 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.892112 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c" event={"ID":"f200e919-64e2-4432-95f2-017382cf700e","Type":"ContainerDied","Data":"af9b9010b7760bf4fbdcf4f1cf8217021d8ac4561f4c05c2bc273d36afb1243d"} Sep 10 19:39:28.900712 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.900691 2578 scope.go:117] "RemoveContainer" containerID="c87c05ce8ebd9eb857810c86c83111d20872a05e3649741cce1bdc60b5d17ac5" Sep 10 19:39:28.901042 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:39:28.901019 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c87c05ce8ebd9eb857810c86c83111d20872a05e3649741cce1bdc60b5d17ac5\": container with ID starting with c87c05ce8ebd9eb857810c86c83111d20872a05e3649741cce1bdc60b5d17ac5 not found: ID does not exist" containerID="c87c05ce8ebd9eb857810c86c83111d20872a05e3649741cce1bdc60b5d17ac5" Sep 10 19:39:28.901138 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.901050 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c87c05ce8ebd9eb857810c86c83111d20872a05e3649741cce1bdc60b5d17ac5"} err="failed to get container status \"c87c05ce8ebd9eb857810c86c83111d20872a05e3649741cce1bdc60b5d17ac5\": rpc error: code = NotFound desc = could not find container \"c87c05ce8ebd9eb857810c86c83111d20872a05e3649741cce1bdc60b5d17ac5\": container with ID starting with c87c05ce8ebd9eb857810c86c83111d20872a05e3649741cce1bdc60b5d17ac5 not found: ID does not exist" Sep 10 19:39:28.901138 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.901066 2578 scope.go:117] "RemoveContainer" containerID="20de745812a33743392d57e3225095a4a63bd553ba204b1a7f1260a3b1f6d4a6" Sep 10 19:39:28.909000 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.908987 2578 scope.go:117] "RemoveContainer" containerID="20de745812a33743392d57e3225095a4a63bd553ba204b1a7f1260a3b1f6d4a6" Sep 10 19:39:28.909262 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:39:28.909245 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20de745812a33743392d57e3225095a4a63bd553ba204b1a7f1260a3b1f6d4a6\": container with ID starting with 20de745812a33743392d57e3225095a4a63bd553ba204b1a7f1260a3b1f6d4a6 not found: ID does not exist" containerID="20de745812a33743392d57e3225095a4a63bd553ba204b1a7f1260a3b1f6d4a6" Sep 10 19:39:28.909305 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.909268 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20de745812a33743392d57e3225095a4a63bd553ba204b1a7f1260a3b1f6d4a6"} err="failed to get container status \"20de745812a33743392d57e3225095a4a63bd553ba204b1a7f1260a3b1f6d4a6\": rpc error: code = NotFound desc = could not find container \"20de745812a33743392d57e3225095a4a63bd553ba204b1a7f1260a3b1f6d4a6\": container with ID starting with 20de745812a33743392d57e3225095a4a63bd553ba204b1a7f1260a3b1f6d4a6 not found: ID does not exist" Sep 10 19:39:28.913067 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.913043 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl"] Sep 10 19:39:28.917017 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.916999 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cbac1f-b4d74585c-24dnl"] Sep 10 19:39:28.926006 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.925983 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c"] Sep 10 19:39:28.928428 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:28.928409 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbac1f-f64964c6b-h847c"] Sep 10 19:39:29.391540 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:29.391509 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cdb8698-93d6-47f2-849f-ed49a69e450d" path="/var/lib/kubelet/pods/8cdb8698-93d6-47f2-849f-ed49a69e450d/volumes" Sep 10 19:39:29.391918 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:29.391850 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f200e919-64e2-4432-95f2-017382cf700e" path="/var/lib/kubelet/pods/f200e919-64e2-4432-95f2-017382cf700e/volumes" Sep 10 19:39:35.110424 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.110389 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx"] Sep 10 19:39:35.110840 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.110740 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b153b56-af63-4517-bbd6-9a53b226146d" containerName="payload-pre-processing" Sep 10 19:39:35.110840 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.110751 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b153b56-af63-4517-bbd6-9a53b226146d" containerName="payload-pre-processing" Sep 10 19:39:35.110840 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.110767 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2846dac5-1949-4a75-a642-78afa838ef89" containerName="payload-processing" Sep 10 19:39:35.110840 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.110772 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2846dac5-1949-4a75-a642-78afa838ef89" containerName="payload-processing" Sep 10 19:39:35.110840 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.110784 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cdb8698-93d6-47f2-849f-ed49a69e450d" containerName="payload-processing" Sep 10 19:39:35.110840 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.110790 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cdb8698-93d6-47f2-849f-ed49a69e450d" containerName="payload-processing" Sep 10 19:39:35.110840 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.110797 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f200e919-64e2-4432-95f2-017382cf700e" containerName="payload-pre-processing" Sep 10 19:39:35.110840 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.110802 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f200e919-64e2-4432-95f2-017382cf700e" containerName="payload-pre-processing" Sep 10 19:39:35.111092 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.110856 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2846dac5-1949-4a75-a642-78afa838ef89" containerName="payload-processing" Sep 10 19:39:35.111092 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.110863 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cdb8698-93d6-47f2-849f-ed49a69e450d" containerName="payload-processing" Sep 10 19:39:35.111092 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.110868 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b153b56-af63-4517-bbd6-9a53b226146d" containerName="payload-pre-processing" Sep 10 19:39:35.111092 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.110874 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f200e919-64e2-4432-95f2-017382cf700e" containerName="payload-pre-processing" Sep 10 19:39:35.115360 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.115343 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx" Sep 10 19:39:35.117619 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.117594 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:39:35.117784 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.117659 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkgt8\"" Sep 10 19:39:35.120588 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.120568 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx"] Sep 10 19:39:35.230693 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.230659 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f0274d-ddd9-4286-be37-4ae2d9638ffc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx\" (UID: \"30f0274d-ddd9-4286-be37-4ae2d9638ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx" Sep 10 19:39:35.331958 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.331922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f0274d-ddd9-4286-be37-4ae2d9638ffc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx\" (UID: \"30f0274d-ddd9-4286-be37-4ae2d9638ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx" Sep 10 19:39:35.332473 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.332455 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f0274d-ddd9-4286-be37-4ae2d9638ffc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx\" (UID: \"30f0274d-ddd9-4286-be37-4ae2d9638ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx" Sep 10 19:39:35.426435 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.426402 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx" Sep 10 19:39:35.696505 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.696478 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx"] Sep 10 19:39:35.698986 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:39:35.698955 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30f0274d_ddd9_4286_be37_4ae2d9638ffc.slice/crio-b577dc57972ef2c80da392d215f9ddb6fa829bfbf419a32610a7e3c6afce2eb8 WatchSource:0}: Error finding container b577dc57972ef2c80da392d215f9ddb6fa829bfbf419a32610a7e3c6afce2eb8: Status 404 returned error can't find the container with id b577dc57972ef2c80da392d215f9ddb6fa829bfbf419a32610a7e3c6afce2eb8 Sep 10 19:39:35.933290 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:35.933259 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx" event={"ID":"30f0274d-ddd9-4286-be37-4ae2d9638ffc","Type":"ContainerStarted","Data":"b577dc57972ef2c80da392d215f9ddb6fa829bfbf419a32610a7e3c6afce2eb8"} Sep 10 19:39:36.939786 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:36.939755 2578 generic.go:356] "Generic (PLEG): container finished" podID="30f0274d-ddd9-4286-be37-4ae2d9638ffc" containerID="9be505b387b19fd0a5768b386db7760683f2997678700cd4f3f61077363b77dd" exitCode=0 Sep 10 19:39:36.940151 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:36.939813 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx" event={"ID":"30f0274d-ddd9-4286-be37-4ae2d9638ffc","Type":"ContainerDied","Data":"9be505b387b19fd0a5768b386db7760683f2997678700cd4f3f61077363b77dd"} Sep 10 19:39:38.076376 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:38.076353 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx" Sep 10 19:39:38.158266 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:38.158241 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f0274d-ddd9-4286-be37-4ae2d9638ffc-maas-api-service-ca\") pod \"30f0274d-ddd9-4286-be37-4ae2d9638ffc\" (UID: \"30f0274d-ddd9-4286-be37-4ae2d9638ffc\") " Sep 10 19:39:38.158609 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:38.158590 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30f0274d-ddd9-4286-be37-4ae2d9638ffc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30f0274d-ddd9-4286-be37-4ae2d9638ffc" (UID: "30f0274d-ddd9-4286-be37-4ae2d9638ffc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:39:38.259785 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:38.259692 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30f0274d-ddd9-4286-be37-4ae2d9638ffc-maas-api-service-ca\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:39:38.950944 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:38.950918 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx" Sep 10 19:39:38.950944 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:38.950926 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx" event={"ID":"30f0274d-ddd9-4286-be37-4ae2d9638ffc","Type":"ContainerDied","Data":"b577dc57972ef2c80da392d215f9ddb6fa829bfbf419a32610a7e3c6afce2eb8"} Sep 10 19:39:38.951152 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:38.950969 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b577dc57972ef2c80da392d215f9ddb6fa829bfbf419a32610a7e3c6afce2eb8" Sep 10 19:39:46.060101 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:46.060027 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx"] Sep 10 19:39:46.063767 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:46.063740 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8dd4a3-6bdwx"] Sep 10 19:39:47.391055 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:39:47.391021 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30f0274d-ddd9-4286-be37-4ae2d9638ffc" path="/var/lib/kubelet/pods/30f0274d-ddd9-4286-be37-4ae2d9638ffc/volumes" Sep 10 19:40:04.174591 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.174559 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w"] Sep 10 19:40:04.175072 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.174794 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" podUID="5ca176fe-4985-4a39-97d4-d3ebaee39ee0" containerName="payload-processing" containerID="cri-o://71ba2cdc6c8ac25e7758d92ab97cc815d482747bcb5a6888371d34725fb6a14c" gracePeriod=30 Sep 10 19:40:04.189011 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.188981 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb"] Sep 10 19:40:04.189258 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.189231 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" podUID="2f48b751-f5b1-4b7c-9c9f-f3598002a8b7" containerName="payload-pre-processing" containerID="cri-o://5a7164a6860a5d2165cb3153ed2c8fe077091a9a03553ac10df99e44190f868f" gracePeriod=30 Sep 10 19:40:04.462544 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.462522 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" Sep 10 19:40:04.465244 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.465224 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" Sep 10 19:40:04.587749 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.587688 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbpnl\" (UniqueName: \"kubernetes.io/projected/5ca176fe-4985-4a39-97d4-d3ebaee39ee0-kube-api-access-jbpnl\") pod \"5ca176fe-4985-4a39-97d4-d3ebaee39ee0\" (UID: \"5ca176fe-4985-4a39-97d4-d3ebaee39ee0\") " Sep 10 19:40:04.587931 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.587777 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj46k\" (UniqueName: \"kubernetes.io/projected/2f48b751-f5b1-4b7c-9c9f-f3598002a8b7-kube-api-access-pj46k\") pod \"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7\" (UID: \"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7\") " Sep 10 19:40:04.587931 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.587834 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f48b751-f5b1-4b7c-9c9f-f3598002a8b7-plugins-config-volume\") pod \"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7\" (UID: \"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7\") " Sep 10 19:40:04.587931 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.587898 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca176fe-4985-4a39-97d4-d3ebaee39ee0-plugins-config-volume\") pod \"5ca176fe-4985-4a39-97d4-d3ebaee39ee0\" (UID: \"5ca176fe-4985-4a39-97d4-d3ebaee39ee0\") " Sep 10 19:40:04.588271 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.588234 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f48b751-f5b1-4b7c-9c9f-f3598002a8b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f48b751-f5b1-4b7c-9c9f-f3598002a8b7" (UID: "2f48b751-f5b1-4b7c-9c9f-f3598002a8b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:40:04.588398 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.588277 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca176fe-4985-4a39-97d4-d3ebaee39ee0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5ca176fe-4985-4a39-97d4-d3ebaee39ee0" (UID: "5ca176fe-4985-4a39-97d4-d3ebaee39ee0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:40:04.590026 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.590005 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f48b751-f5b1-4b7c-9c9f-f3598002a8b7-kube-api-access-pj46k" (OuterVolumeSpecName: "kube-api-access-pj46k") pod "2f48b751-f5b1-4b7c-9c9f-f3598002a8b7" (UID: "2f48b751-f5b1-4b7c-9c9f-f3598002a8b7"). InnerVolumeSpecName "kube-api-access-pj46k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:40:04.590197 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.590183 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca176fe-4985-4a39-97d4-d3ebaee39ee0-kube-api-access-jbpnl" (OuterVolumeSpecName: "kube-api-access-jbpnl") pod "5ca176fe-4985-4a39-97d4-d3ebaee39ee0" (UID: "5ca176fe-4985-4a39-97d4-d3ebaee39ee0"). InnerVolumeSpecName "kube-api-access-jbpnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:40:04.689452 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.689377 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca176fe-4985-4a39-97d4-d3ebaee39ee0-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:40:04.689452 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.689401 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbpnl\" (UniqueName: \"kubernetes.io/projected/5ca176fe-4985-4a39-97d4-d3ebaee39ee0-kube-api-access-jbpnl\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:40:04.689452 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.689410 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj46k\" (UniqueName: \"kubernetes.io/projected/2f48b751-f5b1-4b7c-9c9f-f3598002a8b7-kube-api-access-pj46k\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:40:04.689452 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:04.689420 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f48b751-f5b1-4b7c-9c9f-f3598002a8b7-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:40:05.076311 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.076210 2578 generic.go:356] "Generic (PLEG): container finished" podID="5ca176fe-4985-4a39-97d4-d3ebaee39ee0" containerID="71ba2cdc6c8ac25e7758d92ab97cc815d482747bcb5a6888371d34725fb6a14c" exitCode=0 Sep 10 19:40:05.076311 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.076277 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" Sep 10 19:40:05.076311 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.076298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" event={"ID":"5ca176fe-4985-4a39-97d4-d3ebaee39ee0","Type":"ContainerDied","Data":"71ba2cdc6c8ac25e7758d92ab97cc815d482747bcb5a6888371d34725fb6a14c"} Sep 10 19:40:05.076604 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.076382 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w" event={"ID":"5ca176fe-4985-4a39-97d4-d3ebaee39ee0","Type":"ContainerDied","Data":"598df124c069d13bcaa810dc690ffed359321f32f9d10fb65e6497e4b8baff4e"} Sep 10 19:40:05.076604 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.076407 2578 scope.go:117] "RemoveContainer" containerID="71ba2cdc6c8ac25e7758d92ab97cc815d482747bcb5a6888371d34725fb6a14c" Sep 10 19:40:05.077699 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.077675 2578 generic.go:356] "Generic (PLEG): container finished" podID="2f48b751-f5b1-4b7c-9c9f-f3598002a8b7" containerID="5a7164a6860a5d2165cb3153ed2c8fe077091a9a03553ac10df99e44190f868f" exitCode=0 Sep 10 19:40:05.077810 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.077741 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" Sep 10 19:40:05.077810 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.077765 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" event={"ID":"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7","Type":"ContainerDied","Data":"5a7164a6860a5d2165cb3153ed2c8fe077091a9a03553ac10df99e44190f868f"} Sep 10 19:40:05.077930 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.077815 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb" event={"ID":"2f48b751-f5b1-4b7c-9c9f-f3598002a8b7","Type":"ContainerDied","Data":"ec1a50124669493526e172e2f40f16f577422ff4749f93fa606402b215e40765"} Sep 10 19:40:05.085802 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.085786 2578 scope.go:117] "RemoveContainer" containerID="71ba2cdc6c8ac25e7758d92ab97cc815d482747bcb5a6888371d34725fb6a14c" Sep 10 19:40:05.086137 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:40:05.086090 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ba2cdc6c8ac25e7758d92ab97cc815d482747bcb5a6888371d34725fb6a14c\": container with ID starting with 71ba2cdc6c8ac25e7758d92ab97cc815d482747bcb5a6888371d34725fb6a14c not found: ID does not exist" containerID="71ba2cdc6c8ac25e7758d92ab97cc815d482747bcb5a6888371d34725fb6a14c" Sep 10 19:40:05.086263 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.086126 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ba2cdc6c8ac25e7758d92ab97cc815d482747bcb5a6888371d34725fb6a14c"} err="failed to get container status \"71ba2cdc6c8ac25e7758d92ab97cc815d482747bcb5a6888371d34725fb6a14c\": rpc error: code = NotFound desc = could not find container \"71ba2cdc6c8ac25e7758d92ab97cc815d482747bcb5a6888371d34725fb6a14c\": container with ID starting with 71ba2cdc6c8ac25e7758d92ab97cc815d482747bcb5a6888371d34725fb6a14c not found: ID does not exist" Sep 10 19:40:05.086263 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.086151 2578 scope.go:117] "RemoveContainer" containerID="5a7164a6860a5d2165cb3153ed2c8fe077091a9a03553ac10df99e44190f868f" Sep 10 19:40:05.094792 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.094774 2578 scope.go:117] "RemoveContainer" containerID="5a7164a6860a5d2165cb3153ed2c8fe077091a9a03553ac10df99e44190f868f" Sep 10 19:40:05.095105 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:40:05.095088 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7164a6860a5d2165cb3153ed2c8fe077091a9a03553ac10df99e44190f868f\": container with ID starting with 5a7164a6860a5d2165cb3153ed2c8fe077091a9a03553ac10df99e44190f868f not found: ID does not exist" containerID="5a7164a6860a5d2165cb3153ed2c8fe077091a9a03553ac10df99e44190f868f" Sep 10 19:40:05.095155 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.095112 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7164a6860a5d2165cb3153ed2c8fe077091a9a03553ac10df99e44190f868f"} err="failed to get container status \"5a7164a6860a5d2165cb3153ed2c8fe077091a9a03553ac10df99e44190f868f\": rpc error: code = NotFound desc = could not find container \"5a7164a6860a5d2165cb3153ed2c8fe077091a9a03553ac10df99e44190f868f\": container with ID starting with 5a7164a6860a5d2165cb3153ed2c8fe077091a9a03553ac10df99e44190f868f not found: ID does not exist" Sep 10 19:40:05.108488 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.108468 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w"] Sep 10 19:40:05.114168 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.114151 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4bc8b3-6c8d9757c5-9rk5w"] Sep 10 19:40:05.124197 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.124180 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb"] Sep 10 19:40:05.126199 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.126181 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-4bc8b3-f47d97dbb-slvtb"] Sep 10 19:40:05.390215 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.390182 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f48b751-f5b1-4b7c-9c9f-f3598002a8b7" path="/var/lib/kubelet/pods/2f48b751-f5b1-4b7c-9c9f-f3598002a8b7/volumes" Sep 10 19:40:05.390571 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:05.390529 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ca176fe-4985-4a39-97d4-d3ebaee39ee0" path="/var/lib/kubelet/pods/5ca176fe-4985-4a39-97d4-d3ebaee39ee0/volumes" Sep 10 19:40:56.134592 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.134558 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw"] Sep 10 19:40:56.134998 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.134889 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ca176fe-4985-4a39-97d4-d3ebaee39ee0" containerName="payload-processing" Sep 10 19:40:56.134998 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.134901 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ca176fe-4985-4a39-97d4-d3ebaee39ee0" containerName="payload-processing" Sep 10 19:40:56.134998 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.134911 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f48b751-f5b1-4b7c-9c9f-f3598002a8b7" containerName="payload-pre-processing" Sep 10 19:40:56.134998 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.134916 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f48b751-f5b1-4b7c-9c9f-f3598002a8b7" containerName="payload-pre-processing" Sep 10 19:40:56.134998 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.134930 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30f0274d-ddd9-4286-be37-4ae2d9638ffc" containerName="revoke-keys" Sep 10 19:40:56.134998 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.134937 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30f0274d-ddd9-4286-be37-4ae2d9638ffc" containerName="revoke-keys" Sep 10 19:40:56.134998 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.135002 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30f0274d-ddd9-4286-be37-4ae2d9638ffc" containerName="revoke-keys" Sep 10 19:40:56.135219 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.135010 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f48b751-f5b1-4b7c-9c9f-f3598002a8b7" containerName="payload-pre-processing" Sep 10 19:40:56.135219 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.135015 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ca176fe-4985-4a39-97d4-d3ebaee39ee0" containerName="payload-processing" Sep 10 19:40:56.139277 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.139259 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" Sep 10 19:40:56.141605 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.141581 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-df863b\"" Sep 10 19:40:56.141721 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.141703 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-df863b-dockercfg-rfxpx\"" Sep 10 19:40:56.151744 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.150406 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw"] Sep 10 19:40:56.324840 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.324802 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919dad71-00b0-414a-94ed-4cc0fa1735c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw\" (UID: \"919dad71-00b0-414a-94ed-4cc0fa1735c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" Sep 10 19:40:56.324840 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.324841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh46b\" (UniqueName: \"kubernetes.io/projected/919dad71-00b0-414a-94ed-4cc0fa1735c9-kube-api-access-vh46b\") pod \"payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw\" (UID: \"919dad71-00b0-414a-94ed-4cc0fa1735c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" Sep 10 19:40:56.426474 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.426375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919dad71-00b0-414a-94ed-4cc0fa1735c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw\" (UID: \"919dad71-00b0-414a-94ed-4cc0fa1735c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" Sep 10 19:40:56.426474 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.426427 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh46b\" (UniqueName: \"kubernetes.io/projected/919dad71-00b0-414a-94ed-4cc0fa1735c9-kube-api-access-vh46b\") pod \"payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw\" (UID: \"919dad71-00b0-414a-94ed-4cc0fa1735c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" Sep 10 19:40:56.427161 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.427133 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919dad71-00b0-414a-94ed-4cc0fa1735c9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw\" (UID: \"919dad71-00b0-414a-94ed-4cc0fa1735c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" Sep 10 19:40:56.435229 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.435205 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh46b\" (UniqueName: \"kubernetes.io/projected/919dad71-00b0-414a-94ed-4cc0fa1735c9-kube-api-access-vh46b\") pod \"payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw\" (UID: \"919dad71-00b0-414a-94ed-4cc0fa1735c9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" Sep 10 19:40:56.458062 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.458040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" Sep 10 19:40:56.728746 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:56.728650 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw"] Sep 10 19:40:56.731359 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:40:56.731333 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919dad71_00b0_414a_94ed_4cc0fa1735c9.slice/crio-659b2c4634d5340382a7a4184f9e84911074f72f2bff6f425550ddc6587fcc0c WatchSource:0}: Error finding container 659b2c4634d5340382a7a4184f9e84911074f72f2bff6f425550ddc6587fcc0c: Status 404 returned error can't find the container with id 659b2c4634d5340382a7a4184f9e84911074f72f2bff6f425550ddc6587fcc0c Sep 10 19:40:57.326500 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:57.326461 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" event={"ID":"919dad71-00b0-414a-94ed-4cc0fa1735c9","Type":"ContainerStarted","Data":"c484d54990186e8cc7d8add39477fe4c62269c3c6ed2f9eef1e58e6178ebfb58"} Sep 10 19:40:57.327000 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:57.326511 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" event={"ID":"919dad71-00b0-414a-94ed-4cc0fa1735c9","Type":"ContainerStarted","Data":"659b2c4634d5340382a7a4184f9e84911074f72f2bff6f425550ddc6587fcc0c"} Sep 10 19:40:57.327000 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:57.326540 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" Sep 10 19:40:57.344238 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:40:57.344190 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" podStartSLOduration=1.3441744500000001 podStartE2EDuration="1.34417445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:40:57.341781994 +0000 UTC m=+1506.472283626" watchObservedRunningTime="2026-09-10 19:40:57.34417445 +0000 UTC m=+1506.474676081" Sep 10 19:41:08.333915 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:08.333883 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" Sep 10 19:41:25.000883 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.000848 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q"] Sep 10 19:41:25.004445 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.004423 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" Sep 10 19:41:25.007639 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.007621 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3765cf\"" Sep 10 19:41:25.011201 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.011179 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4"] Sep 10 19:41:25.014812 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.014791 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" Sep 10 19:41:25.015411 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.015392 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q"] Sep 10 19:41:25.017398 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.017378 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3765cf-dockercfg-t2qbm\"" Sep 10 19:41:25.031804 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.031781 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4"] Sep 10 19:41:25.059318 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.059286 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37e38804-c4ab-48af-9ff1-d9f4d9cc53ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4\" (UID: \"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" Sep 10 19:41:25.059449 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.059348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88c8f692-20c7-4751-8df8-72eee50e0ed3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q\" (UID: \"88c8f692-20c7-4751-8df8-72eee50e0ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" Sep 10 19:41:25.059449 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.059386 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmtqg\" (UniqueName: \"kubernetes.io/projected/37e38804-c4ab-48af-9ff1-d9f4d9cc53ab-kube-api-access-bmtqg\") pod \"payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4\" (UID: \"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" Sep 10 19:41:25.059449 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.059415 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpnzk\" (UniqueName: \"kubernetes.io/projected/88c8f692-20c7-4751-8df8-72eee50e0ed3-kube-api-access-bpnzk\") pod \"payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q\" (UID: \"88c8f692-20c7-4751-8df8-72eee50e0ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" Sep 10 19:41:25.160692 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.160661 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88c8f692-20c7-4751-8df8-72eee50e0ed3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q\" (UID: \"88c8f692-20c7-4751-8df8-72eee50e0ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" Sep 10 19:41:25.160894 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.160696 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmtqg\" (UniqueName: \"kubernetes.io/projected/37e38804-c4ab-48af-9ff1-d9f4d9cc53ab-kube-api-access-bmtqg\") pod \"payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4\" (UID: \"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" Sep 10 19:41:25.160894 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.160717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpnzk\" (UniqueName: \"kubernetes.io/projected/88c8f692-20c7-4751-8df8-72eee50e0ed3-kube-api-access-bpnzk\") pod \"payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q\" (UID: \"88c8f692-20c7-4751-8df8-72eee50e0ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" Sep 10 19:41:25.160894 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.160825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37e38804-c4ab-48af-9ff1-d9f4d9cc53ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4\" (UID: \"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" Sep 10 19:41:25.161366 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.161346 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88c8f692-20c7-4751-8df8-72eee50e0ed3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q\" (UID: \"88c8f692-20c7-4751-8df8-72eee50e0ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" Sep 10 19:41:25.161504 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.161486 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37e38804-c4ab-48af-9ff1-d9f4d9cc53ab-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4\" (UID: \"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" Sep 10 19:41:25.169144 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.169111 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmtqg\" (UniqueName: \"kubernetes.io/projected/37e38804-c4ab-48af-9ff1-d9f4d9cc53ab-kube-api-access-bmtqg\") pod \"payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4\" (UID: \"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" Sep 10 19:41:25.169256 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.169111 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpnzk\" (UniqueName: \"kubernetes.io/projected/88c8f692-20c7-4751-8df8-72eee50e0ed3-kube-api-access-bpnzk\") pod \"payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q\" (UID: \"88c8f692-20c7-4751-8df8-72eee50e0ed3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" Sep 10 19:41:25.316047 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.315974 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" Sep 10 19:41:25.324240 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.324222 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" Sep 10 19:41:25.591912 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.591869 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q"] Sep 10 19:41:25.594647 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:41:25.594609 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c8f692_20c7_4751_8df8_72eee50e0ed3.slice/crio-47cf1348fece415e3c1cc63da7989f818254ede93b05b63b739484bf46d4d2ed WatchSource:0}: Error finding container 47cf1348fece415e3c1cc63da7989f818254ede93b05b63b739484bf46d4d2ed: Status 404 returned error can't find the container with id 47cf1348fece415e3c1cc63da7989f818254ede93b05b63b739484bf46d4d2ed Sep 10 19:41:25.603903 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:25.603883 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4"] Sep 10 19:41:25.607140 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:41:25.607120 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e38804_c4ab_48af_9ff1_d9f4d9cc53ab.slice/crio-91d103c69e703d3278324a182ae367d1e4f5d307704f2ee359a8b51d650d777a WatchSource:0}: Error finding container 91d103c69e703d3278324a182ae367d1e4f5d307704f2ee359a8b51d650d777a: Status 404 returned error can't find the container with id 91d103c69e703d3278324a182ae367d1e4f5d307704f2ee359a8b51d650d777a Sep 10 19:41:26.461901 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:26.461861 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" event={"ID":"88c8f692-20c7-4751-8df8-72eee50e0ed3","Type":"ContainerStarted","Data":"b3f87ef0773b51a8ebd0b25d1b47f80485d02ce35c707d8afcdb86afa149c3af"} Sep 10 19:41:26.462360 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:26.461924 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" Sep 10 19:41:26.462360 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:26.461948 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" event={"ID":"88c8f692-20c7-4751-8df8-72eee50e0ed3","Type":"ContainerStarted","Data":"47cf1348fece415e3c1cc63da7989f818254ede93b05b63b739484bf46d4d2ed"} Sep 10 19:41:26.463323 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:26.463298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" event={"ID":"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab","Type":"ContainerStarted","Data":"1c2ef492ad43c6da374e8edafed879298a7f43e605ba50e867bfc1cb965f2ac6"} Sep 10 19:41:26.463439 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:26.463339 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" event={"ID":"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab","Type":"ContainerStarted","Data":"91d103c69e703d3278324a182ae367d1e4f5d307704f2ee359a8b51d650d777a"} Sep 10 19:41:26.463439 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:26.463376 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" Sep 10 19:41:26.481621 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:26.481574 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" podStartSLOduration=2.481559829 podStartE2EDuration="2.481559829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:41:26.479854932 +0000 UTC m=+1535.610356564" watchObservedRunningTime="2026-09-10 19:41:26.481559829 +0000 UTC m=+1535.612061459" Sep 10 19:41:26.495819 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:26.495779 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" podStartSLOduration=2.4957672 podStartE2EDuration="2.4957672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:41:26.495202325 +0000 UTC m=+1535.625703957" watchObservedRunningTime="2026-09-10 19:41:26.4957672 +0000 UTC m=+1535.626268831" Sep 10 19:41:37.470251 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:37.470222 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" Sep 10 19:41:37.470700 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:37.470289 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" Sep 10 19:41:51.542122 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:51.542091 2578 scope.go:117] "RemoveContainer" containerID="edcd3a922dfc98ce222129cc8386a4e0d624715ba5c3665a837bbc4873b89b15" Sep 10 19:41:53.111683 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:53.111645 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wtc2c_73d78e35-5e50-40d2-b50c-ed3ff13f031e/payload-processing/0.log" Sep 10 19:41:53.229417 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:41:53.229389 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4_37e38804-c4ab-48af-9ff1-d9f4d9cc53ab/payload-processing/0.log" Sep 10 19:42:29.338688 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:42:29.338655 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw_919dad71-00b0-414a-94ed-4cc0fa1735c9/payload-processing/0.log" Sep 10 19:42:29.502653 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:42:29.502571 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wtc2c_73d78e35-5e50-40d2-b50c-ed3ff13f031e/payload-processing/0.log" Sep 10 19:42:29.640164 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:42:29.640138 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4_37e38804-c4ab-48af-9ff1-d9f4d9cc53ab/payload-processing/0.log" Sep 10 19:42:59.287344 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:42:59.287266 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz"] Sep 10 19:42:59.290666 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:42:59.290650 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz" Sep 10 19:42:59.298200 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:42:59.298180 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 19:42:59.299687 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:42:59.299672 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkgt8\"" Sep 10 19:42:59.314690 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:42:59.314667 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz"] Sep 10 19:42:59.351176 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:42:59.351151 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5513dd31-eb86-493e-bcf0-db1b6d0b2cae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz\" (UID: \"5513dd31-eb86-493e-bcf0-db1b6d0b2cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz" Sep 10 19:42:59.452077 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:42:59.452046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5513dd31-eb86-493e-bcf0-db1b6d0b2cae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz\" (UID: \"5513dd31-eb86-493e-bcf0-db1b6d0b2cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz" Sep 10 19:42:59.452764 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:42:59.452745 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5513dd31-eb86-493e-bcf0-db1b6d0b2cae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz\" (UID: \"5513dd31-eb86-493e-bcf0-db1b6d0b2cae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz" Sep 10 19:42:59.599880 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:42:59.599818 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz" Sep 10 19:42:59.894463 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:42:59.894398 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz"] Sep 10 19:42:59.897104 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:42:59.897081 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5513dd31_eb86_493e_bcf0_db1b6d0b2cae.slice/crio-3a8ee7f9bd0071c1778e63a2e4a7c540b4cef88acce7a2d2af74488f3c53f565 WatchSource:0}: Error finding container 3a8ee7f9bd0071c1778e63a2e4a7c540b4cef88acce7a2d2af74488f3c53f565: Status 404 returned error can't find the container with id 3a8ee7f9bd0071c1778e63a2e4a7c540b4cef88acce7a2d2af74488f3c53f565 Sep 10 19:43:00.896678 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:00.896643 2578 generic.go:356] "Generic (PLEG): container finished" podID="5513dd31-eb86-493e-bcf0-db1b6d0b2cae" containerID="cfaa50feac759076e0cac1d934566c7c93d54f78bd6343d667ed7564cb6a3153" exitCode=0 Sep 10 19:43:00.897088 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:00.896716 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz" event={"ID":"5513dd31-eb86-493e-bcf0-db1b6d0b2cae","Type":"ContainerDied","Data":"cfaa50feac759076e0cac1d934566c7c93d54f78bd6343d667ed7564cb6a3153"} Sep 10 19:43:00.897088 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:00.896771 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz" event={"ID":"5513dd31-eb86-493e-bcf0-db1b6d0b2cae","Type":"ContainerStarted","Data":"3a8ee7f9bd0071c1778e63a2e4a7c540b4cef88acce7a2d2af74488f3c53f565"} Sep 10 19:43:02.032110 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:02.032085 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz" Sep 10 19:43:02.073426 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:02.073397 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5513dd31-eb86-493e-bcf0-db1b6d0b2cae-maas-api-service-ca\") pod \"5513dd31-eb86-493e-bcf0-db1b6d0b2cae\" (UID: \"5513dd31-eb86-493e-bcf0-db1b6d0b2cae\") " Sep 10 19:43:02.073771 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:02.073712 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5513dd31-eb86-493e-bcf0-db1b6d0b2cae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5513dd31-eb86-493e-bcf0-db1b6d0b2cae" (UID: "5513dd31-eb86-493e-bcf0-db1b6d0b2cae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:43:02.073884 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:02.073808 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5513dd31-eb86-493e-bcf0-db1b6d0b2cae-maas-api-service-ca\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:43:02.907965 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:02.907933 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz" Sep 10 19:43:02.908129 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:02.907926 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz" event={"ID":"5513dd31-eb86-493e-bcf0-db1b6d0b2cae","Type":"ContainerDied","Data":"3a8ee7f9bd0071c1778e63a2e4a7c540b4cef88acce7a2d2af74488f3c53f565"} Sep 10 19:43:02.908129 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:02.908069 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a8ee7f9bd0071c1778e63a2e4a7c540b4cef88acce7a2d2af74488f3c53f565" Sep 10 19:43:05.778489 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:05.778458 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz"] Sep 10 19:43:05.781320 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:05.781295 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8a4013-7qljz"] Sep 10 19:43:07.390597 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:07.390566 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5513dd31-eb86-493e-bcf0-db1b6d0b2cae" path="/var/lib/kubelet/pods/5513dd31-eb86-493e-bcf0-db1b6d0b2cae/volumes" Sep 10 19:43:32.532330 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:32.532287 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw"] Sep 10 19:43:32.532801 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:32.532589 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" podUID="919dad71-00b0-414a-94ed-4cc0fa1735c9" containerName="payload-processing" containerID="cri-o://c484d54990186e8cc7d8add39477fe4c62269c3c6ed2f9eef1e58e6178ebfb58" gracePeriod=30 Sep 10 19:43:32.810573 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:32.810548 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" Sep 10 19:43:32.936156 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:32.936118 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919dad71-00b0-414a-94ed-4cc0fa1735c9-plugins-config-volume\") pod \"919dad71-00b0-414a-94ed-4cc0fa1735c9\" (UID: \"919dad71-00b0-414a-94ed-4cc0fa1735c9\") " Sep 10 19:43:32.936345 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:32.936176 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh46b\" (UniqueName: \"kubernetes.io/projected/919dad71-00b0-414a-94ed-4cc0fa1735c9-kube-api-access-vh46b\") pod \"919dad71-00b0-414a-94ed-4cc0fa1735c9\" (UID: \"919dad71-00b0-414a-94ed-4cc0fa1735c9\") " Sep 10 19:43:32.936605 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:32.936579 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919dad71-00b0-414a-94ed-4cc0fa1735c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "919dad71-00b0-414a-94ed-4cc0fa1735c9" (UID: "919dad71-00b0-414a-94ed-4cc0fa1735c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:43:32.938457 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:32.938432 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919dad71-00b0-414a-94ed-4cc0fa1735c9-kube-api-access-vh46b" (OuterVolumeSpecName: "kube-api-access-vh46b") pod "919dad71-00b0-414a-94ed-4cc0fa1735c9" (UID: "919dad71-00b0-414a-94ed-4cc0fa1735c9"). InnerVolumeSpecName "kube-api-access-vh46b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:43:33.037296 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:33.037246 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919dad71-00b0-414a-94ed-4cc0fa1735c9-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:43:33.037296 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:33.037288 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh46b\" (UniqueName: \"kubernetes.io/projected/919dad71-00b0-414a-94ed-4cc0fa1735c9-kube-api-access-vh46b\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:43:33.055632 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:33.055550 2578 generic.go:356] "Generic (PLEG): container finished" podID="919dad71-00b0-414a-94ed-4cc0fa1735c9" containerID="c484d54990186e8cc7d8add39477fe4c62269c3c6ed2f9eef1e58e6178ebfb58" exitCode=0 Sep 10 19:43:33.055632 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:33.055597 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" Sep 10 19:43:33.055632 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:33.055623 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" event={"ID":"919dad71-00b0-414a-94ed-4cc0fa1735c9","Type":"ContainerDied","Data":"c484d54990186e8cc7d8add39477fe4c62269c3c6ed2f9eef1e58e6178ebfb58"} Sep 10 19:43:33.055920 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:33.055665 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw" event={"ID":"919dad71-00b0-414a-94ed-4cc0fa1735c9","Type":"ContainerDied","Data":"659b2c4634d5340382a7a4184f9e84911074f72f2bff6f425550ddc6587fcc0c"} Sep 10 19:43:33.055920 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:33.055687 2578 scope.go:117] "RemoveContainer" containerID="c484d54990186e8cc7d8add39477fe4c62269c3c6ed2f9eef1e58e6178ebfb58" Sep 10 19:43:33.064409 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:33.064390 2578 scope.go:117] "RemoveContainer" containerID="c484d54990186e8cc7d8add39477fe4c62269c3c6ed2f9eef1e58e6178ebfb58" Sep 10 19:43:33.064666 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:43:33.064644 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c484d54990186e8cc7d8add39477fe4c62269c3c6ed2f9eef1e58e6178ebfb58\": container with ID starting with c484d54990186e8cc7d8add39477fe4c62269c3c6ed2f9eef1e58e6178ebfb58 not found: ID does not exist" containerID="c484d54990186e8cc7d8add39477fe4c62269c3c6ed2f9eef1e58e6178ebfb58" Sep 10 19:43:33.064716 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:33.064676 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c484d54990186e8cc7d8add39477fe4c62269c3c6ed2f9eef1e58e6178ebfb58"} err="failed to get container status \"c484d54990186e8cc7d8add39477fe4c62269c3c6ed2f9eef1e58e6178ebfb58\": rpc error: code = NotFound desc = could not find container \"c484d54990186e8cc7d8add39477fe4c62269c3c6ed2f9eef1e58e6178ebfb58\": container with ID starting with c484d54990186e8cc7d8add39477fe4c62269c3c6ed2f9eef1e58e6178ebfb58 not found: ID does not exist" Sep 10 19:43:33.076669 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:33.076646 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw"] Sep 10 19:43:33.081256 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:33.081237 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-df863b-79c6ff95c5-nstdw"] Sep 10 19:43:33.390800 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:43:33.390769 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="919dad71-00b0-414a-94ed-4cc0fa1735c9" path="/var/lib/kubelet/pods/919dad71-00b0-414a-94ed-4cc0fa1735c9/volumes" Sep 10 19:44:38.606814 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.606781 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4"] Sep 10 19:44:38.607335 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.607045 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" podUID="37e38804-c4ab-48af-9ff1-d9f4d9cc53ab" containerName="payload-processing" containerID="cri-o://1c2ef492ad43c6da374e8edafed879298a7f43e605ba50e867bfc1cb965f2ac6" gracePeriod=30 Sep 10 19:44:38.613553 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.613524 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q"] Sep 10 19:44:38.613766 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.613720 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" podUID="88c8f692-20c7-4751-8df8-72eee50e0ed3" containerName="payload-pre-processing" containerID="cri-o://b3f87ef0773b51a8ebd0b25d1b47f80485d02ce35c707d8afcdb86afa149c3af" gracePeriod=30 Sep 10 19:44:38.858164 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.858101 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" Sep 10 19:44:38.891030 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.891005 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" Sep 10 19:44:38.994566 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.994535 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpnzk\" (UniqueName: \"kubernetes.io/projected/88c8f692-20c7-4751-8df8-72eee50e0ed3-kube-api-access-bpnzk\") pod \"88c8f692-20c7-4751-8df8-72eee50e0ed3\" (UID: \"88c8f692-20c7-4751-8df8-72eee50e0ed3\") " Sep 10 19:44:38.994759 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.994612 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88c8f692-20c7-4751-8df8-72eee50e0ed3-plugins-config-volume\") pod \"88c8f692-20c7-4751-8df8-72eee50e0ed3\" (UID: \"88c8f692-20c7-4751-8df8-72eee50e0ed3\") " Sep 10 19:44:38.994759 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.994631 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37e38804-c4ab-48af-9ff1-d9f4d9cc53ab-plugins-config-volume\") pod \"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab\" (UID: \"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab\") " Sep 10 19:44:38.994759 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.994680 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmtqg\" (UniqueName: \"kubernetes.io/projected/37e38804-c4ab-48af-9ff1-d9f4d9cc53ab-kube-api-access-bmtqg\") pod \"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab\" (UID: \"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab\") " Sep 10 19:44:38.995072 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.995044 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c8f692-20c7-4751-8df8-72eee50e0ed3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "88c8f692-20c7-4751-8df8-72eee50e0ed3" (UID: "88c8f692-20c7-4751-8df8-72eee50e0ed3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:44:38.995072 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.995069 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e38804-c4ab-48af-9ff1-d9f4d9cc53ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37e38804-c4ab-48af-9ff1-d9f4d9cc53ab" (UID: "37e38804-c4ab-48af-9ff1-d9f4d9cc53ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:44:38.996935 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.996913 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c8f692-20c7-4751-8df8-72eee50e0ed3-kube-api-access-bpnzk" (OuterVolumeSpecName: "kube-api-access-bpnzk") pod "88c8f692-20c7-4751-8df8-72eee50e0ed3" (UID: "88c8f692-20c7-4751-8df8-72eee50e0ed3"). InnerVolumeSpecName "kube-api-access-bpnzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:44:38.997015 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:38.996946 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e38804-c4ab-48af-9ff1-d9f4d9cc53ab-kube-api-access-bmtqg" (OuterVolumeSpecName: "kube-api-access-bmtqg") pod "37e38804-c4ab-48af-9ff1-d9f4d9cc53ab" (UID: "37e38804-c4ab-48af-9ff1-d9f4d9cc53ab"). InnerVolumeSpecName "kube-api-access-bmtqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:44:39.095388 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.095355 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmtqg\" (UniqueName: \"kubernetes.io/projected/37e38804-c4ab-48af-9ff1-d9f4d9cc53ab-kube-api-access-bmtqg\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:44:39.095388 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.095387 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpnzk\" (UniqueName: \"kubernetes.io/projected/88c8f692-20c7-4751-8df8-72eee50e0ed3-kube-api-access-bpnzk\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:44:39.095388 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.095396 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88c8f692-20c7-4751-8df8-72eee50e0ed3-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:44:39.095589 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.095405 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37e38804-c4ab-48af-9ff1-d9f4d9cc53ab-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:44:39.372404 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.372371 2578 generic.go:356] "Generic (PLEG): container finished" podID="37e38804-c4ab-48af-9ff1-d9f4d9cc53ab" containerID="1c2ef492ad43c6da374e8edafed879298a7f43e605ba50e867bfc1cb965f2ac6" exitCode=0 Sep 10 19:44:39.372595 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.372412 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" Sep 10 19:44:39.372595 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.372451 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" event={"ID":"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab","Type":"ContainerDied","Data":"1c2ef492ad43c6da374e8edafed879298a7f43e605ba50e867bfc1cb965f2ac6"} Sep 10 19:44:39.372595 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.372498 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4" event={"ID":"37e38804-c4ab-48af-9ff1-d9f4d9cc53ab","Type":"ContainerDied","Data":"91d103c69e703d3278324a182ae367d1e4f5d307704f2ee359a8b51d650d777a"} Sep 10 19:44:39.372595 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.372527 2578 scope.go:117] "RemoveContainer" containerID="1c2ef492ad43c6da374e8edafed879298a7f43e605ba50e867bfc1cb965f2ac6" Sep 10 19:44:39.373855 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.373832 2578 generic.go:356] "Generic (PLEG): container finished" podID="88c8f692-20c7-4751-8df8-72eee50e0ed3" containerID="b3f87ef0773b51a8ebd0b25d1b47f80485d02ce35c707d8afcdb86afa149c3af" exitCode=0 Sep 10 19:44:39.373962 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.373862 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" Sep 10 19:44:39.373962 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.373901 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" event={"ID":"88c8f692-20c7-4751-8df8-72eee50e0ed3","Type":"ContainerDied","Data":"b3f87ef0773b51a8ebd0b25d1b47f80485d02ce35c707d8afcdb86afa149c3af"} Sep 10 19:44:39.373962 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.373952 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q" event={"ID":"88c8f692-20c7-4751-8df8-72eee50e0ed3","Type":"ContainerDied","Data":"47cf1348fece415e3c1cc63da7989f818254ede93b05b63b739484bf46d4d2ed"} Sep 10 19:44:39.381308 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.381154 2578 scope.go:117] "RemoveContainer" containerID="1c2ef492ad43c6da374e8edafed879298a7f43e605ba50e867bfc1cb965f2ac6" Sep 10 19:44:39.381493 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:44:39.381467 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c2ef492ad43c6da374e8edafed879298a7f43e605ba50e867bfc1cb965f2ac6\": container with ID starting with 1c2ef492ad43c6da374e8edafed879298a7f43e605ba50e867bfc1cb965f2ac6 not found: ID does not exist" containerID="1c2ef492ad43c6da374e8edafed879298a7f43e605ba50e867bfc1cb965f2ac6" Sep 10 19:44:39.381569 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.381507 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c2ef492ad43c6da374e8edafed879298a7f43e605ba50e867bfc1cb965f2ac6"} err="failed to get container status \"1c2ef492ad43c6da374e8edafed879298a7f43e605ba50e867bfc1cb965f2ac6\": rpc error: code = NotFound desc = could not find container \"1c2ef492ad43c6da374e8edafed879298a7f43e605ba50e867bfc1cb965f2ac6\": container with ID starting with 1c2ef492ad43c6da374e8edafed879298a7f43e605ba50e867bfc1cb965f2ac6 not found: ID does not exist" Sep 10 19:44:39.381569 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.381528 2578 scope.go:117] "RemoveContainer" containerID="b3f87ef0773b51a8ebd0b25d1b47f80485d02ce35c707d8afcdb86afa149c3af" Sep 10 19:44:39.390424 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.390404 2578 scope.go:117] "RemoveContainer" containerID="b3f87ef0773b51a8ebd0b25d1b47f80485d02ce35c707d8afcdb86afa149c3af" Sep 10 19:44:39.390711 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:44:39.390690 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f87ef0773b51a8ebd0b25d1b47f80485d02ce35c707d8afcdb86afa149c3af\": container with ID starting with b3f87ef0773b51a8ebd0b25d1b47f80485d02ce35c707d8afcdb86afa149c3af not found: ID does not exist" containerID="b3f87ef0773b51a8ebd0b25d1b47f80485d02ce35c707d8afcdb86afa149c3af" Sep 10 19:44:39.390813 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.390738 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f87ef0773b51a8ebd0b25d1b47f80485d02ce35c707d8afcdb86afa149c3af"} err="failed to get container status \"b3f87ef0773b51a8ebd0b25d1b47f80485d02ce35c707d8afcdb86afa149c3af\": rpc error: code = NotFound desc = could not find container \"b3f87ef0773b51a8ebd0b25d1b47f80485d02ce35c707d8afcdb86afa149c3af\": container with ID starting with b3f87ef0773b51a8ebd0b25d1b47f80485d02ce35c707d8afcdb86afa149c3af not found: ID does not exist" Sep 10 19:44:39.396661 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.396637 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4"] Sep 10 19:44:39.402204 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.402184 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3765cf-7d8f568c5-6r8j4"] Sep 10 19:44:39.413608 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.413585 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q"] Sep 10 19:44:39.420007 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:39.419989 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3765cf-6fb74cc7d-mq26q"] Sep 10 19:44:41.390637 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:41.390606 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37e38804-c4ab-48af-9ff1-d9f4d9cc53ab" path="/var/lib/kubelet/pods/37e38804-c4ab-48af-9ff1-d9f4d9cc53ab/volumes" Sep 10 19:44:41.391022 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:41.390958 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c8f692-20c7-4751-8df8-72eee50e0ed3" path="/var/lib/kubelet/pods/88c8f692-20c7-4751-8df8-72eee50e0ed3/volumes" Sep 10 19:44:56.403254 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.403223 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6"] Sep 10 19:44:56.404068 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.404034 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" podUID="4207211f-1f91-4a49-99c3-d443939a4471" containerName="payload-processing" containerID="cri-o://02bc402fb5a9812755b6a4cfc6adec4eca34b915a3749bc06ec86e9bae2ea17b" gracePeriod=30 Sep 10 19:44:56.405551 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.405525 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw"] Sep 10 19:44:56.405930 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.405886 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" podUID="32b343b4-2960-44b5-b701-b6f94ea5c181" containerName="payload-pre-processing" containerID="cri-o://ed89d7880d12fcbc6bcf31e75586334837653f44b5b9bd3d8b0fff944903bbe0" gracePeriod=30 Sep 10 19:44:56.690427 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.690404 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" Sep 10 19:44:56.695862 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.695842 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" Sep 10 19:44:56.750520 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.750494 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207211f-1f91-4a49-99c3-d443939a4471-plugins-config-volume\") pod \"4207211f-1f91-4a49-99c3-d443939a4471\" (UID: \"4207211f-1f91-4a49-99c3-d443939a4471\") " Sep 10 19:44:56.750673 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.750539 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b343b4-2960-44b5-b701-b6f94ea5c181-plugins-config-volume\") pod \"32b343b4-2960-44b5-b701-b6f94ea5c181\" (UID: \"32b343b4-2960-44b5-b701-b6f94ea5c181\") " Sep 10 19:44:56.750673 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.750581 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jwvl\" (UniqueName: \"kubernetes.io/projected/32b343b4-2960-44b5-b701-b6f94ea5c181-kube-api-access-6jwvl\") pod \"32b343b4-2960-44b5-b701-b6f94ea5c181\" (UID: \"32b343b4-2960-44b5-b701-b6f94ea5c181\") " Sep 10 19:44:56.750673 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.750612 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5cgx\" (UniqueName: \"kubernetes.io/projected/4207211f-1f91-4a49-99c3-d443939a4471-kube-api-access-s5cgx\") pod \"4207211f-1f91-4a49-99c3-d443939a4471\" (UID: \"4207211f-1f91-4a49-99c3-d443939a4471\") " Sep 10 19:44:56.751151 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.750934 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4207211f-1f91-4a49-99c3-d443939a4471-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4207211f-1f91-4a49-99c3-d443939a4471" (UID: "4207211f-1f91-4a49-99c3-d443939a4471"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:44:56.751151 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.750992 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b343b4-2960-44b5-b701-b6f94ea5c181-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "32b343b4-2960-44b5-b701-b6f94ea5c181" (UID: "32b343b4-2960-44b5-b701-b6f94ea5c181"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:44:56.752925 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.752899 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4207211f-1f91-4a49-99c3-d443939a4471-kube-api-access-s5cgx" (OuterVolumeSpecName: "kube-api-access-s5cgx") pod "4207211f-1f91-4a49-99c3-d443939a4471" (UID: "4207211f-1f91-4a49-99c3-d443939a4471"). InnerVolumeSpecName "kube-api-access-s5cgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:44:56.753018 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.752956 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b343b4-2960-44b5-b701-b6f94ea5c181-kube-api-access-6jwvl" (OuterVolumeSpecName: "kube-api-access-6jwvl") pod "32b343b4-2960-44b5-b701-b6f94ea5c181" (UID: "32b343b4-2960-44b5-b701-b6f94ea5c181"). InnerVolumeSpecName "kube-api-access-6jwvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:44:56.851381 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.851352 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b343b4-2960-44b5-b701-b6f94ea5c181-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:44:56.851381 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.851375 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jwvl\" (UniqueName: \"kubernetes.io/projected/32b343b4-2960-44b5-b701-b6f94ea5c181-kube-api-access-6jwvl\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:44:56.851381 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.851384 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5cgx\" (UniqueName: \"kubernetes.io/projected/4207211f-1f91-4a49-99c3-d443939a4471-kube-api-access-s5cgx\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:44:56.851572 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:56.851393 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4207211f-1f91-4a49-99c3-d443939a4471-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:44:57.471683 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.471647 2578 generic.go:356] "Generic (PLEG): container finished" podID="32b343b4-2960-44b5-b701-b6f94ea5c181" containerID="ed89d7880d12fcbc6bcf31e75586334837653f44b5b9bd3d8b0fff944903bbe0" exitCode=0 Sep 10 19:44:57.472131 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.471704 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" Sep 10 19:44:57.472131 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.471763 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" event={"ID":"32b343b4-2960-44b5-b701-b6f94ea5c181","Type":"ContainerDied","Data":"ed89d7880d12fcbc6bcf31e75586334837653f44b5b9bd3d8b0fff944903bbe0"} Sep 10 19:44:57.472131 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.471832 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw" event={"ID":"32b343b4-2960-44b5-b701-b6f94ea5c181","Type":"ContainerDied","Data":"1e9aba64383851df21941728efd0f1e9af8e4645f19d594280d388fffac8ee8c"} Sep 10 19:44:57.472131 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.471865 2578 scope.go:117] "RemoveContainer" containerID="ed89d7880d12fcbc6bcf31e75586334837653f44b5b9bd3d8b0fff944903bbe0" Sep 10 19:44:57.473334 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.473228 2578 generic.go:356] "Generic (PLEG): container finished" podID="4207211f-1f91-4a49-99c3-d443939a4471" containerID="02bc402fb5a9812755b6a4cfc6adec4eca34b915a3749bc06ec86e9bae2ea17b" exitCode=0 Sep 10 19:44:57.473334 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.473262 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" event={"ID":"4207211f-1f91-4a49-99c3-d443939a4471","Type":"ContainerDied","Data":"02bc402fb5a9812755b6a4cfc6adec4eca34b915a3749bc06ec86e9bae2ea17b"} Sep 10 19:44:57.473334 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.473288 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" Sep 10 19:44:57.473334 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.473298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6" event={"ID":"4207211f-1f91-4a49-99c3-d443939a4471","Type":"ContainerDied","Data":"975e053e5de4f49285d8801c04474d753757462bf6636feefc936aaa1e42e1de"} Sep 10 19:44:57.481264 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.481241 2578 scope.go:117] "RemoveContainer" containerID="ed89d7880d12fcbc6bcf31e75586334837653f44b5b9bd3d8b0fff944903bbe0" Sep 10 19:44:57.481568 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:44:57.481540 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed89d7880d12fcbc6bcf31e75586334837653f44b5b9bd3d8b0fff944903bbe0\": container with ID starting with ed89d7880d12fcbc6bcf31e75586334837653f44b5b9bd3d8b0fff944903bbe0 not found: ID does not exist" containerID="ed89d7880d12fcbc6bcf31e75586334837653f44b5b9bd3d8b0fff944903bbe0" Sep 10 19:44:57.481665 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.481583 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed89d7880d12fcbc6bcf31e75586334837653f44b5b9bd3d8b0fff944903bbe0"} err="failed to get container status \"ed89d7880d12fcbc6bcf31e75586334837653f44b5b9bd3d8b0fff944903bbe0\": rpc error: code = NotFound desc = could not find container \"ed89d7880d12fcbc6bcf31e75586334837653f44b5b9bd3d8b0fff944903bbe0\": container with ID starting with ed89d7880d12fcbc6bcf31e75586334837653f44b5b9bd3d8b0fff944903bbe0 not found: ID does not exist" Sep 10 19:44:57.481665 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.481604 2578 scope.go:117] "RemoveContainer" containerID="02bc402fb5a9812755b6a4cfc6adec4eca34b915a3749bc06ec86e9bae2ea17b" Sep 10 19:44:57.490268 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.490241 2578 scope.go:117] "RemoveContainer" containerID="02bc402fb5a9812755b6a4cfc6adec4eca34b915a3749bc06ec86e9bae2ea17b" Sep 10 19:44:57.490521 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:44:57.490503 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02bc402fb5a9812755b6a4cfc6adec4eca34b915a3749bc06ec86e9bae2ea17b\": container with ID starting with 02bc402fb5a9812755b6a4cfc6adec4eca34b915a3749bc06ec86e9bae2ea17b not found: ID does not exist" containerID="02bc402fb5a9812755b6a4cfc6adec4eca34b915a3749bc06ec86e9bae2ea17b" Sep 10 19:44:57.490566 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.490527 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02bc402fb5a9812755b6a4cfc6adec4eca34b915a3749bc06ec86e9bae2ea17b"} err="failed to get container status \"02bc402fb5a9812755b6a4cfc6adec4eca34b915a3749bc06ec86e9bae2ea17b\": rpc error: code = NotFound desc = could not find container \"02bc402fb5a9812755b6a4cfc6adec4eca34b915a3749bc06ec86e9bae2ea17b\": container with ID starting with 02bc402fb5a9812755b6a4cfc6adec4eca34b915a3749bc06ec86e9bae2ea17b not found: ID does not exist" Sep 10 19:44:57.496138 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.496117 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw"] Sep 10 19:44:57.499530 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.499511 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-14a8d1-c55cb5658-dn7gw"] Sep 10 19:44:57.512617 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.512592 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6"] Sep 10 19:44:57.520863 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:57.520842 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-14a8d1-5fcc5bf5b5-vhxv6"] Sep 10 19:44:59.390951 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:59.390923 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32b343b4-2960-44b5-b701-b6f94ea5c181" path="/var/lib/kubelet/pods/32b343b4-2960-44b5-b701-b6f94ea5c181/volumes" Sep 10 19:44:59.391314 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:44:59.391269 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4207211f-1f91-4a49-99c3-d443939a4471" path="/var/lib/kubelet/pods/4207211f-1f91-4a49-99c3-d443939a4471/volumes" Sep 10 19:45:00.146272 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146238 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cab4ee-29817825-mr8rg"] Sep 10 19:45:00.146548 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146536 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="919dad71-00b0-414a-94ed-4cc0fa1735c9" containerName="payload-processing" Sep 10 19:45:00.146600 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146550 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="919dad71-00b0-414a-94ed-4cc0fa1735c9" containerName="payload-processing" Sep 10 19:45:00.146600 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146563 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32b343b4-2960-44b5-b701-b6f94ea5c181" containerName="payload-pre-processing" Sep 10 19:45:00.146600 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146569 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="32b343b4-2960-44b5-b701-b6f94ea5c181" containerName="payload-pre-processing" Sep 10 19:45:00.146600 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146576 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5513dd31-eb86-493e-bcf0-db1b6d0b2cae" containerName="revoke-keys" Sep 10 19:45:00.146600 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146582 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5513dd31-eb86-493e-bcf0-db1b6d0b2cae" containerName="revoke-keys" Sep 10 19:45:00.146600 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146590 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c8f692-20c7-4751-8df8-72eee50e0ed3" containerName="payload-pre-processing" Sep 10 19:45:00.146600 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146595 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c8f692-20c7-4751-8df8-72eee50e0ed3" containerName="payload-pre-processing" Sep 10 19:45:00.146838 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146603 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4207211f-1f91-4a49-99c3-d443939a4471" containerName="payload-processing" Sep 10 19:45:00.146838 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146609 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4207211f-1f91-4a49-99c3-d443939a4471" containerName="payload-processing" Sep 10 19:45:00.146838 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146618 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37e38804-c4ab-48af-9ff1-d9f4d9cc53ab" containerName="payload-processing" Sep 10 19:45:00.146838 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146624 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="37e38804-c4ab-48af-9ff1-d9f4d9cc53ab" containerName="payload-processing" Sep 10 19:45:00.146838 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146670 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="919dad71-00b0-414a-94ed-4cc0fa1735c9" containerName="payload-processing" Sep 10 19:45:00.146838 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146677 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5513dd31-eb86-493e-bcf0-db1b6d0b2cae" containerName="revoke-keys" Sep 10 19:45:00.146838 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146682 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4207211f-1f91-4a49-99c3-d443939a4471" containerName="payload-processing" Sep 10 19:45:00.146838 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146688 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="37e38804-c4ab-48af-9ff1-d9f4d9cc53ab" containerName="payload-processing" Sep 10 19:45:00.146838 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146695 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="32b343b4-2960-44b5-b701-b6f94ea5c181" containerName="payload-pre-processing" Sep 10 19:45:00.146838 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.146702 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c8f692-20c7-4751-8df8-72eee50e0ed3" containerName="payload-pre-processing" Sep 10 19:45:00.150970 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.150954 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cab4ee-29817825-mr8rg" Sep 10 19:45:00.153856 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.153831 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wkgt8\"" Sep 10 19:45:00.158412 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.158386 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cab4ee-29817825-mr8rg"] Sep 10 19:45:00.162311 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.162294 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cab4ee-29817825-mr8rg" Sep 10 19:45:00.431714 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.431643 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cab4ee-29817825-mr8rg"] Sep 10 19:45:00.434903 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:45:00.434874 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2b4165_7a8e_4968_9a27_cffef3cbac8c.slice/crio-57ff4e005ce9287444e0d80e7f2f8e080acc8cbe265978a02fe142ffa711c885 WatchSource:0}: Error finding container 57ff4e005ce9287444e0d80e7f2f8e080acc8cbe265978a02fe142ffa711c885: Status 404 returned error can't find the container with id 57ff4e005ce9287444e0d80e7f2f8e080acc8cbe265978a02fe142ffa711c885 Sep 10 19:45:00.491520 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:00.491486 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cab4ee-29817825-mr8rg" event={"ID":"de2b4165-7a8e-4968-9a27-cffef3cbac8c","Type":"ContainerStarted","Data":"57ff4e005ce9287444e0d80e7f2f8e080acc8cbe265978a02fe142ffa711c885"} Sep 10 19:45:01.505100 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:01.505062 2578 generic.go:356] "Generic (PLEG): container finished" podID="de2b4165-7a8e-4968-9a27-cffef3cbac8c" containerID="e14548eb4921e9ee9db510a2f607b0c5f2608495780302fb95c09c07a974564d" exitCode=0 Sep 10 19:45:01.505495 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:01.505114 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cab4ee-29817825-mr8rg" event={"ID":"de2b4165-7a8e-4968-9a27-cffef3cbac8c","Type":"ContainerDied","Data":"e14548eb4921e9ee9db510a2f607b0c5f2608495780302fb95c09c07a974564d"} Sep 10 19:45:02.641656 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:02.641634 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cab4ee-29817825-mr8rg" Sep 10 19:45:03.515707 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:03.515672 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cab4ee-29817825-mr8rg" event={"ID":"de2b4165-7a8e-4968-9a27-cffef3cbac8c","Type":"ContainerDied","Data":"57ff4e005ce9287444e0d80e7f2f8e080acc8cbe265978a02fe142ffa711c885"} Sep 10 19:45:03.515707 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:03.515695 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cab4ee-29817825-mr8rg" Sep 10 19:45:03.515912 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:03.515745 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57ff4e005ce9287444e0d80e7f2f8e080acc8cbe265978a02fe142ffa711c885" Sep 10 19:45:14.431085 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:14.431049 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cab4ee-29817825-mr8rg"] Sep 10 19:45:14.432872 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:14.432847 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-cab4ee-29817825-mr8rg"] Sep 10 19:45:14.457276 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:14.457245 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz"] Sep 10 19:45:14.457490 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:14.457460 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" podUID="db04029d-705d-459d-b42d-03df6c5f845e" containerName="payload-processing" containerID="cri-o://c7d15dfc99f4fd947e01a31776a5df637e91423e5eb6e8c8d154106247aadb16" gracePeriod=30 Sep 10 19:45:14.711091 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:14.711065 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" Sep 10 19:45:14.804057 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:14.804027 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db04029d-705d-459d-b42d-03df6c5f845e-plugins-config-volume\") pod \"db04029d-705d-459d-b42d-03df6c5f845e\" (UID: \"db04029d-705d-459d-b42d-03df6c5f845e\") " Sep 10 19:45:14.804522 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:14.804087 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nltzz\" (UniqueName: \"kubernetes.io/projected/db04029d-705d-459d-b42d-03df6c5f845e-kube-api-access-nltzz\") pod \"db04029d-705d-459d-b42d-03df6c5f845e\" (UID: \"db04029d-705d-459d-b42d-03df6c5f845e\") " Sep 10 19:45:14.804522 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:14.804433 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db04029d-705d-459d-b42d-03df6c5f845e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db04029d-705d-459d-b42d-03df6c5f845e" (UID: "db04029d-705d-459d-b42d-03df6c5f845e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 19:45:14.806717 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:14.806693 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db04029d-705d-459d-b42d-03df6c5f845e-kube-api-access-nltzz" (OuterVolumeSpecName: "kube-api-access-nltzz") pod "db04029d-705d-459d-b42d-03df6c5f845e" (UID: "db04029d-705d-459d-b42d-03df6c5f845e"). InnerVolumeSpecName "kube-api-access-nltzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:45:14.904940 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:14.904911 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db04029d-705d-459d-b42d-03df6c5f845e-plugins-config-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:45:14.904940 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:14.904936 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nltzz\" (UniqueName: \"kubernetes.io/projected/db04029d-705d-459d-b42d-03df6c5f845e-kube-api-access-nltzz\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:45:15.390033 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:15.390000 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de2b4165-7a8e-4968-9a27-cffef3cbac8c" path="/var/lib/kubelet/pods/de2b4165-7a8e-4968-9a27-cffef3cbac8c/volumes" Sep 10 19:45:15.568691 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:15.568653 2578 generic.go:356] "Generic (PLEG): container finished" podID="db04029d-705d-459d-b42d-03df6c5f845e" containerID="c7d15dfc99f4fd947e01a31776a5df637e91423e5eb6e8c8d154106247aadb16" exitCode=0 Sep 10 19:45:15.569108 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:15.568690 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" event={"ID":"db04029d-705d-459d-b42d-03df6c5f845e","Type":"ContainerDied","Data":"c7d15dfc99f4fd947e01a31776a5df637e91423e5eb6e8c8d154106247aadb16"} Sep 10 19:45:15.569108 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:15.568698 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" Sep 10 19:45:15.569108 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:15.568797 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz" event={"ID":"db04029d-705d-459d-b42d-03df6c5f845e","Type":"ContainerDied","Data":"591fdf2fb99b47a03518a48bae0112aff9c9797625dcf7270949897e5b4361d9"} Sep 10 19:45:15.569108 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:15.568823 2578 scope.go:117] "RemoveContainer" containerID="c7d15dfc99f4fd947e01a31776a5df637e91423e5eb6e8c8d154106247aadb16" Sep 10 19:45:15.577242 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:15.577225 2578 scope.go:117] "RemoveContainer" containerID="c7d15dfc99f4fd947e01a31776a5df637e91423e5eb6e8c8d154106247aadb16" Sep 10 19:45:15.577520 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:45:15.577502 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d15dfc99f4fd947e01a31776a5df637e91423e5eb6e8c8d154106247aadb16\": container with ID starting with c7d15dfc99f4fd947e01a31776a5df637e91423e5eb6e8c8d154106247aadb16 not found: ID does not exist" containerID="c7d15dfc99f4fd947e01a31776a5df637e91423e5eb6e8c8d154106247aadb16" Sep 10 19:45:15.577585 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:15.577527 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d15dfc99f4fd947e01a31776a5df637e91423e5eb6e8c8d154106247aadb16"} err="failed to get container status \"c7d15dfc99f4fd947e01a31776a5df637e91423e5eb6e8c8d154106247aadb16\": rpc error: code = NotFound desc = could not find container \"c7d15dfc99f4fd947e01a31776a5df637e91423e5eb6e8c8d154106247aadb16\": container with ID starting with c7d15dfc99f4fd947e01a31776a5df637e91423e5eb6e8c8d154106247aadb16 not found: ID does not exist" Sep 10 19:45:15.583931 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:15.583911 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz"] Sep 10 19:45:15.587468 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:15.587447 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-cab4ee-8448dc77cc-fpmmz"] Sep 10 19:45:17.391478 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:17.391443 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db04029d-705d-459d-b42d-03df6c5f845e" path="/var/lib/kubelet/pods/db04029d-705d-459d-b42d-03df6c5f845e/volumes" Sep 10 19:45:51.591461 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:45:51.591425 2578 scope.go:117] "RemoveContainer" containerID="9be505b387b19fd0a5768b386db7760683f2997678700cd4f3f61077363b77dd" Sep 10 19:47:59.083964 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:47:59.083926 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn"] Sep 10 19:47:59.084412 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:47:59.084176 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" podUID="24f1bba8-35ab-4369-a146-582ceef19c0f" containerName="manager" containerID="cri-o://0ab6a87b637349d9a21edcba038c8a607e8abf3df33ae139ea3f0fd97d3892c0" gracePeriod=10 Sep 10 19:47:59.525854 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:47:59.525826 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" Sep 10 19:47:59.637070 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:47:59.636999 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9wqp\" (UniqueName: \"kubernetes.io/projected/24f1bba8-35ab-4369-a146-582ceef19c0f-kube-api-access-k9wqp\") pod \"24f1bba8-35ab-4369-a146-582ceef19c0f\" (UID: \"24f1bba8-35ab-4369-a146-582ceef19c0f\") " Sep 10 19:47:59.637206 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:47:59.637108 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24f1bba8-35ab-4369-a146-582ceef19c0f-extensions-socket-volume\") pod \"24f1bba8-35ab-4369-a146-582ceef19c0f\" (UID: \"24f1bba8-35ab-4369-a146-582ceef19c0f\") " Sep 10 19:47:59.637471 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:47:59.637447 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f1bba8-35ab-4369-a146-582ceef19c0f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "24f1bba8-35ab-4369-a146-582ceef19c0f" (UID: "24f1bba8-35ab-4369-a146-582ceef19c0f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 19:47:59.639334 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:47:59.639307 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f1bba8-35ab-4369-a146-582ceef19c0f-kube-api-access-k9wqp" (OuterVolumeSpecName: "kube-api-access-k9wqp") pod "24f1bba8-35ab-4369-a146-582ceef19c0f" (UID: "24f1bba8-35ab-4369-a146-582ceef19c0f"). InnerVolumeSpecName "kube-api-access-k9wqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 19:47:59.738042 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:47:59.738014 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/24f1bba8-35ab-4369-a146-582ceef19c0f-extensions-socket-volume\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:47:59.738042 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:47:59.738040 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9wqp\" (UniqueName: \"kubernetes.io/projected/24f1bba8-35ab-4369-a146-582ceef19c0f-kube-api-access-k9wqp\") on node \"ip-10-0-138-71.ec2.internal\" DevicePath \"\"" Sep 10 19:48:00.315423 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:00.315386 2578 generic.go:356] "Generic (PLEG): container finished" podID="24f1bba8-35ab-4369-a146-582ceef19c0f" containerID="0ab6a87b637349d9a21edcba038c8a607e8abf3df33ae139ea3f0fd97d3892c0" exitCode=0 Sep 10 19:48:00.315893 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:00.315427 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" Sep 10 19:48:00.315893 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:00.315465 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" event={"ID":"24f1bba8-35ab-4369-a146-582ceef19c0f","Type":"ContainerDied","Data":"0ab6a87b637349d9a21edcba038c8a607e8abf3df33ae139ea3f0fd97d3892c0"} Sep 10 19:48:00.315893 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:00.315513 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn" event={"ID":"24f1bba8-35ab-4369-a146-582ceef19c0f","Type":"ContainerDied","Data":"bcbd6177c4a07ff958671cbae16443d50bd0a235d067c0a738aa44bb71c90448"} Sep 10 19:48:00.315893 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:00.315534 2578 scope.go:117] "RemoveContainer" containerID="0ab6a87b637349d9a21edcba038c8a607e8abf3df33ae139ea3f0fd97d3892c0" Sep 10 19:48:00.324810 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:00.324794 2578 scope.go:117] "RemoveContainer" containerID="0ab6a87b637349d9a21edcba038c8a607e8abf3df33ae139ea3f0fd97d3892c0" Sep 10 19:48:00.325068 ip-10-0-138-71 kubenswrapper[2578]: E0910 19:48:00.325050 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ab6a87b637349d9a21edcba038c8a607e8abf3df33ae139ea3f0fd97d3892c0\": container with ID starting with 0ab6a87b637349d9a21edcba038c8a607e8abf3df33ae139ea3f0fd97d3892c0 not found: ID does not exist" containerID="0ab6a87b637349d9a21edcba038c8a607e8abf3df33ae139ea3f0fd97d3892c0" Sep 10 19:48:00.325127 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:00.325080 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ab6a87b637349d9a21edcba038c8a607e8abf3df33ae139ea3f0fd97d3892c0"} err="failed to get container status \"0ab6a87b637349d9a21edcba038c8a607e8abf3df33ae139ea3f0fd97d3892c0\": rpc error: code = NotFound desc = could not find container \"0ab6a87b637349d9a21edcba038c8a607e8abf3df33ae139ea3f0fd97d3892c0\": container with ID starting with 0ab6a87b637349d9a21edcba038c8a607e8abf3df33ae139ea3f0fd97d3892c0 not found: ID does not exist" Sep 10 19:48:00.337329 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:00.337306 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn"] Sep 10 19:48:00.340785 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:00.340758 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kn8qn"] Sep 10 19:48:01.389809 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:01.389763 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24f1bba8-35ab-4369-a146-582ceef19c0f" path="/var/lib/kubelet/pods/24f1bba8-35ab-4369-a146-582ceef19c0f/volumes" Sep 10 19:48:59.746921 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.746884 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4"] Sep 10 19:48:59.747320 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.747297 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db04029d-705d-459d-b42d-03df6c5f845e" containerName="payload-processing" Sep 10 19:48:59.747320 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.747312 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="db04029d-705d-459d-b42d-03df6c5f845e" containerName="payload-processing" Sep 10 19:48:59.747391 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.747325 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f1bba8-35ab-4369-a146-582ceef19c0f" containerName="manager" Sep 10 19:48:59.747391 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.747330 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f1bba8-35ab-4369-a146-582ceef19c0f" containerName="manager" Sep 10 19:48:59.747391 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.747353 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2b4165-7a8e-4968-9a27-cffef3cbac8c" containerName="cleanup" Sep 10 19:48:59.747391 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.747359 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2b4165-7a8e-4968-9a27-cffef3cbac8c" containerName="cleanup" Sep 10 19:48:59.747519 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.747411 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2b4165-7a8e-4968-9a27-cffef3cbac8c" containerName="cleanup" Sep 10 19:48:59.747519 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.747423 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="24f1bba8-35ab-4369-a146-582ceef19c0f" containerName="manager" Sep 10 19:48:59.747519 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.747433 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="db04029d-705d-459d-b42d-03df6c5f845e" containerName="payload-processing" Sep 10 19:48:59.750243 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.750227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4" Sep 10 19:48:59.753638 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.753618 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 10 19:48:59.754769 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.754752 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2r8zh\"" Sep 10 19:48:59.754938 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.754919 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 10 19:48:59.764049 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.764028 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4"] Sep 10 19:48:59.818181 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.818159 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztpt7\" (UniqueName: \"kubernetes.io/projected/853a942e-b26b-4daa-a67b-a3acb34a6da0-kube-api-access-ztpt7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5rcg4\" (UID: \"853a942e-b26b-4daa-a67b-a3acb34a6da0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4" Sep 10 19:48:59.818334 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.818313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/853a942e-b26b-4daa-a67b-a3acb34a6da0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5rcg4\" (UID: \"853a942e-b26b-4daa-a67b-a3acb34a6da0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4" Sep 10 19:48:59.919004 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.918977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/853a942e-b26b-4daa-a67b-a3acb34a6da0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5rcg4\" (UID: \"853a942e-b26b-4daa-a67b-a3acb34a6da0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4" Sep 10 19:48:59.919101 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.919008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztpt7\" (UniqueName: \"kubernetes.io/projected/853a942e-b26b-4daa-a67b-a3acb34a6da0-kube-api-access-ztpt7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5rcg4\" (UID: \"853a942e-b26b-4daa-a67b-a3acb34a6da0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4" Sep 10 19:48:59.919317 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.919299 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/853a942e-b26b-4daa-a67b-a3acb34a6da0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5rcg4\" (UID: \"853a942e-b26b-4daa-a67b-a3acb34a6da0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4" Sep 10 19:48:59.929715 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:48:59.929698 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztpt7\" (UniqueName: \"kubernetes.io/projected/853a942e-b26b-4daa-a67b-a3acb34a6da0-kube-api-access-ztpt7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5rcg4\" (UID: \"853a942e-b26b-4daa-a67b-a3acb34a6da0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4" Sep 10 19:49:00.059575 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:49:00.059504 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4" Sep 10 19:49:00.336635 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:49:00.336611 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4"] Sep 10 19:49:00.341910 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:49:00.341835 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853a942e_b26b_4daa_a67b_a3acb34a6da0.slice/crio-60bb41a9a8757fce8100750ecbdce0e53cf514de819f860a17c08052b6d9b136 WatchSource:0}: Error finding container 60bb41a9a8757fce8100750ecbdce0e53cf514de819f860a17c08052b6d9b136: Status 404 returned error can't find the container with id 60bb41a9a8757fce8100750ecbdce0e53cf514de819f860a17c08052b6d9b136 Sep 10 19:49:00.578805 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:49:00.578774 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4" event={"ID":"853a942e-b26b-4daa-a67b-a3acb34a6da0","Type":"ContainerStarted","Data":"9ecc2d2c90be82c23740cf4185e8b977f606aafcf97e2787e83199dbf45e496b"} Sep 10 19:49:00.578974 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:49:00.578822 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4" event={"ID":"853a942e-b26b-4daa-a67b-a3acb34a6da0","Type":"ContainerStarted","Data":"60bb41a9a8757fce8100750ecbdce0e53cf514de819f860a17c08052b6d9b136"} Sep 10 19:49:00.578974 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:49:00.578852 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4" Sep 10 19:49:00.599385 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:49:00.599296 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4" podStartSLOduration=1.5992804889999999 podStartE2EDuration="1.599280489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 19:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:49:00.597916653 +0000 UTC m=+1989.728418288" watchObservedRunningTime="2026-09-10 19:49:00.599280489 +0000 UTC m=+1989.729782119" Sep 10 19:49:11.584858 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:49:11.584827 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5rcg4" Sep 10 19:49:51.641797 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:49:51.641757 2578 scope.go:117] "RemoveContainer" containerID="cfaa50feac759076e0cac1d934566c7c93d54f78bd6343d667ed7564cb6a3153" Sep 10 19:50:55.202464 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:50:55.202437 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d477566c-rcth8_e66ef0f5-d717-48c0-a395-a5001bf4b377/copy-manifests/0.log" Sep 10 19:50:55.215124 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:50:55.215099 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d477566c-rcth8_e66ef0f5-d717-48c0-a395-a5001bf4b377/manager/0.log" Sep 10 19:50:55.783081 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:50:55.783052 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817810-88dgw_4fe9602d-36f1-48a2-bfaf-cea8b2b039b7/cleanup/0.log" Sep 10 19:50:57.213069 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:50:57.213036 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-5rcg4_853a942e-b26b-4daa-a67b-a3acb34a6da0/manager/0.log" Sep 10 19:50:57.890626 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:50:57.890591 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p596x_eb01b46a-180c-4163-9c01-c219b7b2ada9/discovery/0.log" Sep 10 19:50:58.482176 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:50:58.482149 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wtc2c_73d78e35-5e50-40d2-b50c-ed3ff13f031e/payload-processing/0.log" Sep 10 19:50:58.602155 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:50:58.602130 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c595d49d-6t78r_63f79296-d132-41d4-a902-feeffb16437a/router/0.log" Sep 10 19:50:59.303039 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:50:59.303010 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-h9dp2_887a4e2a-40b5-448a-82f1-b28c7677a674/main/0.log" Sep 10 19:50:59.419214 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:50:59.419188 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fxhkql_4cbe0ddb-33c9-4b7e-a1c6-4381dc29c8ca/main/0.log" Sep 10 19:50:59.527300 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:50:59.527275 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-w944g_df7cf0ab-0e3a-41c8-8b30-733b3643c7a4/main/0.log" Sep 10 19:51:08.805889 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:08.805855 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xdr42_0f3cd70c-826e-4955-8532-a9d444835bb8/global-pull-secret-syncer/0.log" Sep 10 19:51:08.904039 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:08.904006 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bzjwd_f48a3422-593c-4bed-9a93-991b18e65cab/konnectivity-agent/0.log" Sep 10 19:51:08.980781 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:08.980748 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-71.ec2.internal_d41321aee93f122eed6c1597c05c23f6/haproxy/0.log" Sep 10 19:51:13.976182 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:13.976148 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-5rcg4_853a942e-b26b-4daa-a67b-a3acb34a6da0/manager/0.log" Sep 10 19:51:15.847074 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:15.847000 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-j69hg_bc714527-42ce-4346-bfc8-3e782c031a63/cluster-monitoring-operator/0.log" Sep 10 19:51:16.111268 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:16.111194 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9492w_98770593-3fdd-4367-9d4b-8859a01db7b3/node-exporter/0.log" Sep 10 19:51:16.133189 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:16.133161 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9492w_98770593-3fdd-4367-9d4b-8859a01db7b3/kube-rbac-proxy/0.log" Sep 10 19:51:16.157333 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:16.157307 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9492w_98770593-3fdd-4367-9d4b-8859a01db7b3/init-textfile/0.log" Sep 10 19:51:16.607279 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:16.607251 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77bb685b6d-n57b9_fd4f2784-48b4-4050-b12a-b882ce154c73/telemeter-client/0.log" Sep 10 19:51:16.629272 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:16.629246 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77bb685b6d-n57b9_fd4f2784-48b4-4050-b12a-b882ce154c73/reload/0.log" Sep 10 19:51:16.649691 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:16.649660 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-77bb685b6d-n57b9_fd4f2784-48b4-4050-b12a-b882ce154c73/kube-rbac-proxy/0.log" Sep 10 19:51:18.068758 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.068715 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-gfzqc_22fced2d-122a-4a5b-b955-0729b7e2ba6d/networking-console-plugin/0.log" Sep 10 19:51:18.573031 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.572991 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm"] Sep 10 19:51:18.576494 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.576470 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.581822 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.581796 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5hbck\"/\"kube-root-ca.crt\"" Sep 10 19:51:18.582603 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.582580 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5hbck\"/\"default-dockercfg-r6dbn\"" Sep 10 19:51:18.584463 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.584441 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5hbck\"/\"openshift-service-ca.crt\"" Sep 10 19:51:18.650374 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.650339 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm"] Sep 10 19:51:18.694634 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.694595 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwvzp\" (UniqueName: \"kubernetes.io/projected/7a0e603b-ccdd-4733-a162-0ea2279da993-kube-api-access-dwvzp\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.694634 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.694646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a0e603b-ccdd-4733-a162-0ea2279da993-podres\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.694921 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.694685 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a0e603b-ccdd-4733-a162-0ea2279da993-sys\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.694921 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.694764 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a0e603b-ccdd-4733-a162-0ea2279da993-lib-modules\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.694921 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.694831 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a0e603b-ccdd-4733-a162-0ea2279da993-proc\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.795655 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.795614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwvzp\" (UniqueName: \"kubernetes.io/projected/7a0e603b-ccdd-4733-a162-0ea2279da993-kube-api-access-dwvzp\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.795655 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.795655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a0e603b-ccdd-4733-a162-0ea2279da993-podres\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.795917 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.795683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a0e603b-ccdd-4733-a162-0ea2279da993-sys\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.795917 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.795762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a0e603b-ccdd-4733-a162-0ea2279da993-lib-modules\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.795917 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.795833 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a0e603b-ccdd-4733-a162-0ea2279da993-proc\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.795917 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.795844 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a0e603b-ccdd-4733-a162-0ea2279da993-sys\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.795917 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.795855 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a0e603b-ccdd-4733-a162-0ea2279da993-podres\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.795917 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.795900 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a0e603b-ccdd-4733-a162-0ea2279da993-proc\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.796241 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.795961 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a0e603b-ccdd-4733-a162-0ea2279da993-lib-modules\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.808441 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.808406 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwvzp\" (UniqueName: \"kubernetes.io/projected/7a0e603b-ccdd-4733-a162-0ea2279da993-kube-api-access-dwvzp\") pod \"perf-node-gather-daemonset-4mntm\" (UID: \"7a0e603b-ccdd-4733-a162-0ea2279da993\") " pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:18.887948 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:18.887891 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:19.159791 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:19.159763 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm"] Sep 10 19:51:19.161813 ip-10-0-138-71 kubenswrapper[2578]: W0910 19:51:19.161782 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7a0e603b_ccdd_4733_a162_0ea2279da993.slice/crio-3d8d1b2a5c3cfd95b148d43570668c7c81a6138b5f8f87a8c380682efc01f755 WatchSource:0}: Error finding container 3d8d1b2a5c3cfd95b148d43570668c7c81a6138b5f8f87a8c380682efc01f755: Status 404 returned error can't find the container with id 3d8d1b2a5c3cfd95b148d43570668c7c81a6138b5f8f87a8c380682efc01f755 Sep 10 19:51:19.199484 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:19.199429 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" event={"ID":"7a0e603b-ccdd-4733-a162-0ea2279da993","Type":"ContainerStarted","Data":"3d8d1b2a5c3cfd95b148d43570668c7c81a6138b5f8f87a8c380682efc01f755"} Sep 10 19:51:20.205600 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:20.205567 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" event={"ID":"7a0e603b-ccdd-4733-a162-0ea2279da993","Type":"ContainerStarted","Data":"7c0a61fe315c0716452975dc9ed53e06524ecbff27fd14af99ff1a31954f16a8"} Sep 10 19:51:20.206012 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:20.205699 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:20.223983 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:20.223932 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" podStartSLOduration=2.223915978 podStartE2EDuration="2.223915978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 19:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 19:51:20.221835593 +0000 UTC m=+2129.352337223" watchObservedRunningTime="2026-09-10 19:51:20.223915978 +0000 UTC m=+2129.354417608" Sep 10 19:51:21.293167 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:21.293138 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6hxvz_97404b03-33d9-40fa-ad37-25daa6ef148d/dns/0.log" Sep 10 19:51:21.318164 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:21.318133 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6hxvz_97404b03-33d9-40fa-ad37-25daa6ef148d/kube-rbac-proxy/0.log" Sep 10 19:51:21.460001 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:21.459969 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nqdhp_8776ab82-f99c-483e-8001-54e1d88a778d/dns-node-resolver/0.log" Sep 10 19:51:22.082558 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:22.082527 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4b2dk_0358ef59-3ffa-429f-b980-45ce19f19dab/node-ca/0.log" Sep 10 19:51:23.313553 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:23.313519 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p596x_eb01b46a-180c-4163-9c01-c219b7b2ada9/discovery/0.log" Sep 10 19:51:23.478911 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:23.478872 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wtc2c_73d78e35-5e50-40d2-b50c-ed3ff13f031e/payload-processing/0.log" Sep 10 19:51:23.501368 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:23.501336 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c595d49d-6t78r_63f79296-d132-41d4-a902-feeffb16437a/router/0.log" Sep 10 19:51:24.448076 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:24.448045 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jshvk_980c5d3f-72f2-4efb-812c-58f2b9ea0d14/serve-healthcheck-canary/0.log" Sep 10 19:51:24.929760 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:24.929721 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-j4bl4_de135e89-c44c-41c2-badb-87b930765e4a/insights-operator/0.log" Sep 10 19:51:24.930473 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:24.930455 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-j4bl4_de135e89-c44c-41c2-badb-87b930765e4a/insights-operator/1.log" Sep 10 19:51:24.955319 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:24.955294 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d2dlk_ab4d6cda-d5ab-493a-b7aa-0838c7457a55/kube-rbac-proxy/0.log" Sep 10 19:51:24.976695 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:24.976662 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d2dlk_ab4d6cda-d5ab-493a-b7aa-0838c7457a55/exporter/0.log" Sep 10 19:51:24.998198 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:24.998164 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d2dlk_ab4d6cda-d5ab-493a-b7aa-0838c7457a55/extractor/0.log" Sep 10 19:51:26.220237 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:26.220205 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5hbck/perf-node-gather-daemonset-4mntm" Sep 10 19:51:28.173886 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:28.173856 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d477566c-rcth8_e66ef0f5-d717-48c0-a395-a5001bf4b377/manager/0.log" Sep 10 19:51:28.213254 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:28.213225 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d477566c-rcth8_e66ef0f5-d717-48c0-a395-a5001bf4b377/copy-manifests/0.log" Sep 10 19:51:29.443036 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:29.443008 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p596x_eb01b46a-180c-4163-9c01-c219b7b2ada9/discovery/0.log" Sep 10 19:51:29.518183 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:29.518161 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-b5db5bfb7-f9k4s_bacc8f17-9307-4b60-bedc-e236fa0281ac/manager/0.log" Sep 10 19:51:29.571079 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:29.571053 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wtc2c_73d78e35-5e50-40d2-b50c-ed3ff13f031e/payload-processing/0.log" Sep 10 19:51:29.584186 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:29.584165 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c595d49d-6t78r_63f79296-d132-41d4-a902-feeffb16437a/router/0.log" Sep 10 19:51:31.455887 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:31.455848 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-p596x_eb01b46a-180c-4163-9c01-c219b7b2ada9/discovery/0.log" Sep 10 19:51:31.639486 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:31.639456 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-wtc2c_73d78e35-5e50-40d2-b50c-ed3ff13f031e/payload-processing/0.log" Sep 10 19:51:31.654865 ip-10-0-138-71 kubenswrapper[2578]: I0910 19:51:31.654837 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7c595d49d-6t78r_63f79296-d132-41d4-a902-feeffb16437a/router/0.log"