Sep 08 14:14:31.975303 ip-10-0-141-155 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 08 14:14:31.975318 ip-10-0-141-155 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 08 14:14:31.975327 ip-10-0-141-155 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 08 14:14:31.975625 ip-10-0-141-155 systemd[1]: Failed to start Kubernetes Kubelet. Sep 08 14:14:42.176508 ip-10-0-141-155 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 08 14:14:42.176529 ip-10-0-141-155 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 2f339c571d654362a3c2696634d508c9 -- Sep 08 14:16:27.598008 ip-10-0-141-155 systemd[1]: Starting Kubernetes Kubelet... Sep 08 14:16:28.019370 ip-10-0-141-155 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 14:16:28.019370 ip-10-0-141-155 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 08 14:16:28.019370 ip-10-0-141-155 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 14:16:28.019370 ip-10-0-141-155 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 08 14:16:28.019370 ip-10-0-141-155 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 08 14:16:28.020543 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.020449 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 08 14:16:28.023569 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023554 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 14:16:28.023569 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023569 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023573 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023576 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023579 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023584 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023588 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023591 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023594 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023597 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023599 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023602 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023604 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023607 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023609 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023611 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023614 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023616 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023619 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023621 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023624 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 14:16:28.023633 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023628 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023631 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023633 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023636 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023639 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023642 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023645 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023647 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023650 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023652 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023655 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023659 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023663 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023666 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023668 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023671 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023673 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023676 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023678 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 14:16:28.024110 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023681 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023683 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023686 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023689 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023691 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023694 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023696 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023698 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023701 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023703 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023706 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023708 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023710 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023713 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023715 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023718 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023720 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023723 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023744 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023749 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 14:16:28.024573 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023753 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023757 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023759 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023762 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023765 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023769 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023771 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023774 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023777 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023780 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023782 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023785 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023787 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023790 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023792 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023795 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023797 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023800 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023802 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023805 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 14:16:28.025120 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023807 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023809 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023812 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023815 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023818 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023820 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023823 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023826 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023828 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023831 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023834 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023836 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023839 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023843 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023846 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023849 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023852 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023854 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.023857 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 14:16:28.025588 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025500 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025511 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025514 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025517 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025520 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025524 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025527 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025530 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025533 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025535 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025538 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025541 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025543 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025545 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025548 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025550 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025553 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025556 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025558 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025561 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 14:16:28.026049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025563 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025566 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025569 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025571 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025573 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025576 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025578 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025581 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025583 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025586 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025590 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025594 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025598 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025601 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025604 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025606 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025609 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025611 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025614 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 14:16:28.026637 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025616 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025619 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025621 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025624 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025627 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025630 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025632 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025634 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025637 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025639 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025642 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025645 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025647 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025650 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025653 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025656 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025658 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025661 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025663 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 14:16:28.027349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025666 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025668 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025671 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025673 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025676 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025678 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025680 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025684 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025686 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025689 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025691 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025694 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025696 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025698 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025701 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025703 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025706 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025708 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025710 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025713 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 14:16:28.027853 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025716 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025719 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025739 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025742 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025744 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025747 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025750 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025752 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025755 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025757 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025761 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025765 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025767 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025770 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025773 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025775 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025778 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025781 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025784 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 08 14:16:28.028332 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025787 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.025791 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026157 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026161 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026164 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026167 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026170 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026172 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026175 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026178 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026180 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026182 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026185 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026188 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026190 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026195 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026197 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026200 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026202 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026205 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 08 14:16:28.028825 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026207 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026210 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026213 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026215 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026218 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026220 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026222 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026225 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026227 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026231 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026235 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026238 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026241 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026277 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026548 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026556 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026560 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026565 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026570 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026574 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 14:16:28.029308 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026579 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026583 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026587 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026592 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026596 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026602 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026606 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026618 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026623 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026627 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026632 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026636 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026640 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026649 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026656 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026661 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026665 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026670 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026679 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 14:16:28.029811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026683 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026688 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026692 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026696 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026701 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026705 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026720 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026745 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026751 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026755 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026760 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026764 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026772 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026776 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026780 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026784 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026789 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026793 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026798 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026802 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 14:16:28.030293 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026806 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026853 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026860 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026863 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026872 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026877 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026881 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026885 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026889 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026893 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026897 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026901 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026905 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026910 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026914 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026918 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026925 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026935 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026940 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 14:16:28.030783 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026945 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026978 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.026983 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.027263 2578 flags.go:64] FLAG: --address="0.0.0.0" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.027317 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028165 2578 flags.go:64] FLAG: --anonymous-auth="true" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028273 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028279 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028282 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028287 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028291 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028294 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028297 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028300 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028303 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028306 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028309 2578 flags.go:64] FLAG: --cgroup-root="" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028312 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028315 2578 flags.go:64] FLAG: --client-ca-file="" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028318 2578 flags.go:64] FLAG: --cloud-provider="external" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028321 2578 flags.go:64] FLAG: --cluster-dns="[]" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028331 2578 flags.go:64] FLAG: --cluster-domain="" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028334 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028337 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 08 14:16:28.031241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028340 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028344 2578 flags.go:64] FLAG: --container-log-max-files="5" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028347 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028350 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028353 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028356 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028359 2578 flags.go:64] FLAG: --contention-profiling="false" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028363 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028366 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028368 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028371 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028375 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028379 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028383 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028386 2578 flags.go:64] FLAG: --enable-load-reader="false" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028389 2578 flags.go:64] FLAG: --enable-server="true" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028392 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028397 2578 flags.go:64] FLAG: --event-burst="100" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028400 2578 flags.go:64] FLAG: --event-qps="50" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028403 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028406 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028409 2578 flags.go:64] FLAG: --eviction-hard="" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028413 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028415 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028418 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 08 14:16:28.031812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028421 2578 flags.go:64] FLAG: --eviction-soft="" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028424 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028427 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028430 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028432 2578 flags.go:64] FLAG: --experimental-mounter-path="" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028435 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028438 2578 flags.go:64] FLAG: --fail-swap-on="true" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028441 2578 flags.go:64] FLAG: --feature-gates="" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028444 2578 flags.go:64] FLAG: --file-check-frequency="20s" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028447 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028451 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028454 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028457 2578 flags.go:64] FLAG: --healthz-port="10248" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028460 2578 flags.go:64] FLAG: --help="false" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028463 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028466 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028468 2578 flags.go:64] FLAG: --http-check-frequency="20s" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028471 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028475 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028480 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028483 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028485 2578 flags.go:64] FLAG: --image-service-endpoint="" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028488 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 08 14:16:28.032400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028491 2578 flags.go:64] FLAG: --kube-api-burst="100" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028494 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028497 2578 flags.go:64] FLAG: --kube-api-qps="50" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028500 2578 flags.go:64] FLAG: --kube-reserved="" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028502 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028506 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028509 2578 flags.go:64] FLAG: --kubelet-cgroups="" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028512 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028514 2578 flags.go:64] FLAG: --lock-file="" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028517 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028520 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028523 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028528 2578 flags.go:64] FLAG: --log-json-split-stream="false" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028531 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028541 2578 flags.go:64] FLAG: --log-text-split-stream="false" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028546 2578 flags.go:64] FLAG: --logging-format="text" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028548 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028552 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028554 2578 flags.go:64] FLAG: --manifest-url="" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028558 2578 flags.go:64] FLAG: --manifest-url-header="" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028563 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028566 2578 flags.go:64] FLAG: --max-open-files="1000000" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028570 2578 flags.go:64] FLAG: --max-pods="110" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028573 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028576 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 08 14:16:28.033010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028594 2578 flags.go:64] FLAG: --memory-manager-policy="None" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028598 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028602 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028605 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028608 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028616 2578 flags.go:64] FLAG: --node-status-max-images="50" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028620 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028623 2578 flags.go:64] FLAG: --oom-score-adj="-999" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028626 2578 flags.go:64] FLAG: --pod-cidr="" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028629 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028634 2578 flags.go:64] FLAG: --pod-manifest-path="" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028636 2578 flags.go:64] FLAG: --pod-max-pids="-1" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028639 2578 flags.go:64] FLAG: --pods-per-core="0" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028642 2578 flags.go:64] FLAG: --port="10250" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028645 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028648 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0114dc9c47befc9d5" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028651 2578 flags.go:64] FLAG: --qos-reserved="" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028654 2578 flags.go:64] FLAG: --read-only-port="10255" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028657 2578 flags.go:64] FLAG: --register-node="true" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028659 2578 flags.go:64] FLAG: --register-with-taints="" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028671 2578 flags.go:64] FLAG: --registry-burst="10" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028674 2578 flags.go:64] FLAG: --registry-qps="5" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028677 2578 flags.go:64] FLAG: --reserved-cpus="" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028680 2578 flags.go:64] FLAG: --reserved-memory="" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028683 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 08 14:16:28.033645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028686 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028691 2578 flags.go:64] FLAG: --rotate-certificates="false" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028694 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028697 2578 flags.go:64] FLAG: --runonce="false" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028699 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028702 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028705 2578 flags.go:64] FLAG: --seccomp-default="false" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028708 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028710 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028713 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028716 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028720 2578 flags.go:64] FLAG: --storage-driver-password="root" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028723 2578 flags.go:64] FLAG: --storage-driver-secure="false" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028741 2578 flags.go:64] FLAG: --storage-driver-table="stats" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028745 2578 flags.go:64] FLAG: --storage-driver-user="root" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028748 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028751 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028753 2578 flags.go:64] FLAG: --system-cgroups="" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028756 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028763 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028766 2578 flags.go:64] FLAG: --tls-cert-file="" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028769 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028777 2578 flags.go:64] FLAG: --tls-min-version="" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028779 2578 flags.go:64] FLAG: --tls-private-key-file="" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028783 2578 flags.go:64] FLAG: --topology-manager-policy="none" Sep 08 14:16:28.034279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028785 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028788 2578 flags.go:64] FLAG: --topology-manager-scope="container" Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028791 2578 flags.go:64] FLAG: --v="2" Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028801 2578 flags.go:64] FLAG: --version="false" Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028805 2578 flags.go:64] FLAG: --vmodule="" Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028808 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.028811 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028925 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028930 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028933 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028936 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028939 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028943 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028945 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028948 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028951 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028954 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028957 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028961 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028964 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028966 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028969 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 14:16:28.034889 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028972 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028974 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028978 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028981 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028984 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028988 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028991 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028994 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.028997 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029000 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029002 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029005 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029008 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029010 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029019 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029021 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029024 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029026 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029030 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 14:16:28.035463 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029032 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029035 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029037 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029039 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029042 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029044 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029047 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029049 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029052 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029056 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029058 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029061 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029063 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029066 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029068 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029071 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029073 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029076 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029078 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029081 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 14:16:28.035951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029083 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029086 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029089 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029091 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029093 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029096 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029098 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029100 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029108 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029111 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029114 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029117 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029119 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029122 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029124 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029127 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029129 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029132 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029134 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 14:16:28.036472 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029136 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029140 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029147 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029150 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029153 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029156 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029158 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029161 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029163 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029166 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029168 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029171 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029173 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029176 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029178 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029181 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029183 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029186 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029189 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 14:16:28.036951 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029191 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029193 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029196 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029204 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029206 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029210 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.029213 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.029218 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.036397 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.036413 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.037341 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.037350 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037404 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037408 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037412 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 14:16:28.037461 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037415 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037417 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037420 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037423 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037426 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037429 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037431 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037434 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037436 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037439 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037443 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037447 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037450 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037452 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037455 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037457 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037460 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037462 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037465 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037468 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 14:16:28.037875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037470 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037473 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037476 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037478 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037481 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037483 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037487 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037490 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037492 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037495 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037497 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037500 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037502 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037504 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037507 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037510 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037513 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037515 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037518 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 14:16:28.038371 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037522 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037526 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037529 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037531 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037533 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037536 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037538 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037541 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037544 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037546 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037548 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037551 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037554 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037556 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037558 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037561 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037563 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037565 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037568 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037570 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 14:16:28.038858 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037573 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037576 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037578 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037581 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037583 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037585 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037588 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037591 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037594 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037597 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037599 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037602 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037604 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037606 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037609 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037611 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037614 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037617 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037620 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 14:16:28.039333 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037624 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037627 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037629 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037632 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037634 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037636 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037639 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037641 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037643 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037646 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037648 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037651 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037653 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037655 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037658 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037661 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037663 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 14:16:28.039818 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037666 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.037670 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037793 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037799 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037804 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037808 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037811 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037833 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037837 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037840 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037843 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037845 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037848 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037851 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037854 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 08 14:16:28.040227 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037857 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037860 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037862 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037864 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037867 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037869 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037872 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037874 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037877 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037879 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037882 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037884 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037887 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037890 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037892 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037895 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037898 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037900 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037903 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037905 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 08 14:16:28.040605 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037907 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037910 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037913 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037916 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037920 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037922 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037925 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037927 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037929 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037932 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037934 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037937 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037939 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037942 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037944 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037946 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037949 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037951 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037954 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 08 14:16:28.041189 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037956 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037958 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037961 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037963 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037965 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037968 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037970 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037972 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037975 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037978 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037982 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037985 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037988 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037991 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037994 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.037997 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038000 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038003 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038005 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038009 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 08 14:16:28.041692 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038011 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038014 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038017 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038019 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038022 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038025 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038027 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038030 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038032 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038035 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038037 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038039 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038042 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038044 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038047 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038049 2578 feature_gate.go:328] unrecognized feature gate: Example Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038052 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038055 2578 feature_gate.go:328] unrecognized feature gate: Example2 Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038057 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 08 14:16:28.042357 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038059 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 08 14:16:28.042831 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038062 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 08 14:16:28.042831 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038064 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 08 14:16:28.042831 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038067 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 08 14:16:28.042831 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038069 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 08 14:16:28.042831 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038072 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 08 14:16:28.042831 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038075 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 08 14:16:28.042831 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:28.038077 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 08 14:16:28.042831 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.038081 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 08 14:16:28.042831 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.038185 2578 server.go:962] "Client rotation is on, will bootstrap in background" Sep 08 14:16:28.042831 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.041120 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 08 14:16:28.042831 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.042050 2578 server.go:1019] "Starting client certificate rotation" Sep 08 14:16:28.042831 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.042166 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 14:16:28.042831 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.042209 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 08 14:16:28.044552 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.044523 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 14:16:28.045945 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.045926 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 08 14:16:28.055295 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.055273 2578 log.go:25] "Validated CRI v1 runtime API" Sep 08 14:16:28.059116 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.059099 2578 log.go:25] "Validated CRI v1 image API" Sep 08 14:16:28.059961 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.059945 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 08 14:16:28.063533 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.063514 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d6924767-45be-4aed-8917-a3ead1afe9fb:/dev/nvme0n1p3 e0a90fd7-1fa7-4464-a9f2-e6a3dbbc3d22:/dev/nvme0n1p4] Sep 08 14:16:28.063594 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.063532 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 08 14:16:28.068693 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.068590 2578 manager.go:216] Machine: {Timestamp:2026-09-08 14:16:28.067226474 +0000 UTC m=+0.351543579 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100151 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24e772683839d3c7ab4ab3e115413c SystemUUID:ec24e772-6838-39d3-c7ab-4ab3e115413c BootID:2f339c57-1d65-4362-a3c2-696634d508c9 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2c:1f:f9:2d:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2c:1f:f9:2d:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:73:5b:3f:bc:8b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 08 14:16:28.068693 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.068691 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 08 14:16:28.068837 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.068819 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 08 14:16:28.070151 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.070125 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 08 14:16:28.070287 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.070153 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-155.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 08 14:16:28.070908 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.070897 2578 topology_manager.go:138] "Creating topology manager with none policy" Sep 08 14:16:28.070940 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.070911 2578 container_manager_linux.go:309] "Creating device plugin manager" Sep 08 14:16:28.070940 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.070919 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 14:16:28.071766 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.071756 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 08 14:16:28.072993 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.072983 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 08 14:16:28.076077 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.076065 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 08 14:16:28.076183 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.076174 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 08 14:16:28.076216 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.076206 2578 kubelet.go:486] "Attempting to sync node with API server" Sep 08 14:16:28.076242 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.076229 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 08 14:16:28.076274 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.076248 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 08 14:16:28.076274 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.076261 2578 kubelet.go:398] "Adding apiserver pod source" Sep 08 14:16:28.076274 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.076273 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 08 14:16:28.076602 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.076591 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 14:16:28.076659 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.076610 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 08 14:16:28.077568 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.077549 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 14:16:28.078967 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.078942 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 08 14:16:28.081072 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.081057 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 08 14:16:28.081136 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.081092 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 08 14:16:28.082010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.081993 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 08 14:16:28.082078 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.082015 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 08 14:16:28.082078 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.082021 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 08 14:16:28.082078 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.082027 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 08 14:16:28.082078 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.082034 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 08 14:16:28.082078 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.082040 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 08 14:16:28.082238 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.082170 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 08 14:16:28.082238 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.082177 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 08 14:16:28.082238 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.082194 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 08 14:16:28.082238 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.082202 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 08 14:16:28.082238 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.082210 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 08 14:16:28.082238 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.082218 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 08 14:16:28.082970 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.082962 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 08 14:16:28.083003 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.082971 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 08 14:16:28.086771 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.086757 2578 server.go:1268] "Started kubelet" Sep 08 14:16:28.086855 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.086834 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 08 14:16:28.086965 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.086886 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 08 14:16:28.086965 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.086943 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 08 14:16:28.089980 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.089941 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 08 14:16:28.090325 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.090303 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-155.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 08 14:16:28.090412 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.090397 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 08 14:16:28.090706 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.090688 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 08 14:16:28.091211 ip-10-0-141-155 systemd[1]: Started Kubernetes Kubelet. Sep 08 14:16:28.092789 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.092077 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 08 14:16:28.092789 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.092415 2578 server.go:310] "Adding debug handlers to kubelet server" Sep 08 14:16:28.096222 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.096204 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 08 14:16:28.096693 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.095672 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-155.ec2.internal.18d35e248dc6d50e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-155.ec2.internal,UID:ip-10-0-141-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-155.ec2.internal,},FirstTimestamp:2026-09-08 14:16:28.086670606 +0000 UTC m=+0.370987712,LastTimestamp:2026-09-08 14:16:28.086670606 +0000 UTC m=+0.370987712,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-155.ec2.internal,}" Sep 08 14:16:28.097628 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.097615 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 08 14:16:28.097722 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.097701 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found" Sep 08 14:16:28.098388 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.098368 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 08 14:16:28.098388 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.098372 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 08 14:16:28.098522 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.098397 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 08 14:16:28.098522 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.098448 2578 reconstruct.go:97] "Volume reconstruction finished" Sep 08 14:16:28.098522 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.098458 2578 reconciler.go:29] "Reconciler: start to sync state" Sep 08 14:16:28.098522 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.098480 2578 factory.go:55] Registering systemd factory Sep 08 14:16:28.098522 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.098493 2578 factory.go:223] Registration of the systemd container factory successfully Sep 08 14:16:28.098700 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.098673 2578 factory.go:153] Registering CRI-O factory Sep 08 14:16:28.098700 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.098684 2578 factory.go:223] Registration of the crio container factory successfully Sep 08 14:16:28.098773 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.098750 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 08 14:16:28.098773 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.098767 2578 factory.go:103] Registering Raw factory Sep 08 14:16:28.098834 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.098777 2578 manager.go:1196] Started watching for new ooms in manager Sep 08 14:16:28.099080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.099070 2578 manager.go:349] Starting recovery of all containers Sep 08 14:16:28.100099 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.100077 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-155.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 08 14:16:28.100198 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.100158 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 08 14:16:28.103477 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.103457 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mgf7v" Sep 08 14:16:28.106373 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.106358 2578 manager.go:354] Recovery completed Sep 08 14:16:28.110771 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.110753 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 14:16:28.111995 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.111945 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 08 14:16:28.112392 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.112369 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mgf7v" Sep 08 14:16:28.113076 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.113056 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory" Sep 08 14:16:28.113137 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.113091 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 14:16:28.113137 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.113105 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID" Sep 08 14:16:28.113668 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.113651 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 08 14:16:28.113668 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.113667 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 08 14:16:28.113836 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.113678 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 14:16:28.113836 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.113682 2578 state_mem.go:36] "Initialized new in-memory state store" Sep 08 14:16:28.114616 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.114597 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory" Sep 08 14:16:28.114678 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.114627 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 14:16:28.114678 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.114640 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID" Sep 08 14:16:28.114802 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.114791 2578 policy_none.go:49] "None policy: Start" Sep 08 14:16:28.114861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.114809 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 08 14:16:28.114861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.114823 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 08 14:16:28.115437 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.115016 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-155.ec2.internal.18d35e248f59be39 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-155.ec2.internal,UID:ip-10-0-141-155.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-155.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-155.ec2.internal,},FirstTimestamp:2026-09-08 14:16:28.113075769 +0000 UTC m=+0.397392875,LastTimestamp:2026-09-08 14:16:28.113075769 +0000 UTC m=+0.397392875,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-155.ec2.internal,}" Sep 08 14:16:28.115886 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.115866 2578 policy_none.go:47] "Start" Sep 08 14:16:28.162346 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.162325 2578 manager.go:337] "Starting Device Plugin manager" Sep 08 14:16:28.178360 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.162369 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 08 14:16:28.178360 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.162382 2578 server.go:87] "Starting device plugin registration server" Sep 08 14:16:28.178360 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.162615 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 08 14:16:28.178360 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.162629 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 08 14:16:28.178360 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.162720 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 08 14:16:28.178360 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.162868 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 08 14:16:28.178360 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.162877 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 08 14:16:28.178360 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.163376 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 08 14:16:28.178360 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.163420 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-155.ec2.internal\" not found" Sep 08 14:16:28.234771 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.234751 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 08 14:16:28.234889 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.234799 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 08 14:16:28.234889 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.234822 2578 kubelet.go:2448] "Starting kubelet main sync loop" Sep 08 14:16:28.234889 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.234860 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 08 14:16:28.237801 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.237784 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 14:16:28.263093 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.263076 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 14:16:28.263962 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.263943 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory" Sep 08 14:16:28.264037 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.263984 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 14:16:28.264037 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.263999 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID" Sep 08 14:16:28.264037 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.264027 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.273288 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.273244 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.273288 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.273265 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-155.ec2.internal\": node \"ip-10-0-141-155.ec2.internal\" not found" Sep 08 14:16:28.289916 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.289895 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found" Sep 08 14:16:28.335247 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.335218 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal"] Sep 08 14:16:28.335325 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.335303 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 14:16:28.336815 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.336791 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory" Sep 08 14:16:28.336894 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.336825 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 14:16:28.336894 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.336842 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID" Sep 08 14:16:28.338176 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.338161 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 14:16:28.338295 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.338276 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.338347 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.338313 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 14:16:28.338861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.338843 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory" Sep 08 14:16:28.338861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.338857 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory" Sep 08 14:16:28.338982 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.338877 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 14:16:28.338982 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.338879 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 14:16:28.338982 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.338885 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID" Sep 08 14:16:28.338982 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.338890 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID" Sep 08 14:16:28.340064 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.340051 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.340126 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.340074 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 08 14:16:28.340702 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.340687 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientMemory" Sep 08 14:16:28.340826 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.340714 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasNoDiskPressure" Sep 08 14:16:28.340826 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.340742 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeHasSufficientPID" Sep 08 14:16:28.372367 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.372346 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-155.ec2.internal\" not found" node="ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.376899 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.376883 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-155.ec2.internal\" not found" node="ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.389969 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.389949 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found" Sep 08 14:16:28.400320 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.400299 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40496547e3481087fc874b11f52284a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal\" (UID: \"40496547e3481087fc874b11f52284a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.400410 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.400322 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/308f44f9483500741181dba63fa91da4-config\") pod \"kube-apiserver-proxy-ip-10-0-141-155.ec2.internal\" (UID: \"308f44f9483500741181dba63fa91da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.400410 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.400339 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40496547e3481087fc874b11f52284a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal\" (UID: \"40496547e3481087fc874b11f52284a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.490536 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.490510 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found" Sep 08 14:16:28.500837 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.500815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/308f44f9483500741181dba63fa91da4-config\") pod \"kube-apiserver-proxy-ip-10-0-141-155.ec2.internal\" (UID: \"308f44f9483500741181dba63fa91da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.500893 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.500841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40496547e3481087fc874b11f52284a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal\" (UID: \"40496547e3481087fc874b11f52284a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.500893 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.500858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40496547e3481087fc874b11f52284a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal\" (UID: \"40496547e3481087fc874b11f52284a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.500957 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.500927 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40496547e3481087fc874b11f52284a9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal\" (UID: \"40496547e3481087fc874b11f52284a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.500957 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.500932 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/308f44f9483500741181dba63fa91da4-config\") pod \"kube-apiserver-proxy-ip-10-0-141-155.ec2.internal\" (UID: \"308f44f9483500741181dba63fa91da4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.501023 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.500932 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/40496547e3481087fc874b11f52284a9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal\" (UID: \"40496547e3481087fc874b11f52284a9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.591313 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.591248 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found" Sep 08 14:16:28.675781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.675722 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.679998 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.679585 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal" Sep 08 14:16:28.692143 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.692126 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found" Sep 08 14:16:28.792772 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.792743 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found" Sep 08 14:16:28.893357 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:28.893282 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-155.ec2.internal\" not found" Sep 08 14:16:28.950902 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.950876 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 14:16:28.998292 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:28.998271 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal" Sep 08 14:16:29.009942 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.009925 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 14:16:29.011966 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.011954 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" Sep 08 14:16:29.025507 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.025490 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 08 14:16:29.042644 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.042627 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 08 14:16:29.042773 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.042753 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 14:16:29.042814 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.042777 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 08 14:16:29.054018 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.054003 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 14:16:29.076879 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.076855 2578 apiserver.go:52] "Watching apiserver" Sep 08 14:16:29.083804 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.083434 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 08 14:16:29.085466 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.085444 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-6kt9z","kube-system/global-pull-secret-syncer-hsh78","openshift-multus/multus-wrvcw","openshift-multus/network-metrics-daemon-998rf","openshift-network-diagnostics/network-check-target-4njj8","openshift-network-operator/iptables-alerter-pxmln","openshift-ovn-kubernetes/ovnkube-node-tlbbx","kube-system/konnectivity-agent-tpxxk","kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2","openshift-cluster-node-tuning-operator/tuned-vnl5t","openshift-dns/node-resolver-xrx9d","openshift-image-registry/node-ca-hlpbr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal"] Sep 08 14:16:29.087297 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.087283 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.088781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.088762 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hlpbr" Sep 08 14:16:29.089675 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.089652 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 08 14:16:29.089675 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.089665 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 08 14:16:29.089839 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.089704 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 08 14:16:29.089839 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.089745 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 08 14:16:29.089839 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.089755 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8bjnm\"" Sep 08 14:16:29.089839 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.089661 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 08 14:16:29.089988 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.089898 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 08 14:16:29.090494 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.090135 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.090939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.090920 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 08 14:16:29.091012 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.090952 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 08 14:16:29.091012 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.090968 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 08 14:16:29.091120 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.091106 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzjrl\"" Sep 08 14:16:29.091665 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.091649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:29.091752 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.091714 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:29.093059 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.093040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:29.093149 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.093106 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:29.094498 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.094482 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pxmln" Sep 08 14:16:29.095807 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.095791 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tpxxk" Sep 08 14:16:29.096143 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.096120 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 08 14:16:29.096143 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.096139 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 08 14:16:29.096298 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.096168 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 08 14:16:29.096298 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.096183 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 08 14:16:29.096407 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.096301 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 08 14:16:29.096407 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.096361 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f9jmp\"" Sep 08 14:16:29.096587 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.096573 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gcdkf\"" Sep 08 14:16:29.096721 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.096697 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 08 14:16:29.096845 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.096827 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 08 14:16:29.096921 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.096908 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 08 14:16:29.097283 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.097271 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.097980 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.097965 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jt4sg\"" Sep 08 14:16:29.098288 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.098259 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 08 14:16:29.098397 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.098340 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 08 14:16:29.098467 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.098452 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:29.098515 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.098501 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:29.099392 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.099376 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 08 14:16:29.099483 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.099399 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 08 14:16:29.099483 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.099426 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jdmbq\"" Sep 08 14:16:29.100021 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.099991 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.101412 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.101389 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.101891 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.101873 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 08 14:16:29.102121 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.102102 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 08 14:16:29.103490 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.102605 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 08 14:16:29.103490 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.102834 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7cx8f\"" Sep 08 14:16:29.103490 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.102962 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xrx9d" Sep 08 14:16:29.104743 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.104463 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 08 14:16:29.104743 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.104662 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 08 14:16:29.104915 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.104732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-sys-fs\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.104915 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.104824 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk4cc\" (UniqueName: \"kubernetes.io/projected/df4f4a97-b500-41f7-bc90-dca2520927f6-kube-api-access-jk4cc\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.104915 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.104837 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zvvrz\"" Sep 08 14:16:29.104915 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.104864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-slash\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.104915 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.104903 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-node-log\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.105176 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105024 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-cni-bin\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.105176 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105061 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5bf364-23e4-42d1-b576-92465114d3c8-system-cni-dir\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.105176 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d5bf364-23e4-42d1-b576-92465114d3c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.105323 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8d5w\" (UniqueName: \"kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w\") pod \"network-check-target-4njj8\" (UID: \"43ad5ec2-69c5-4c3a-a997-adb20472335a\") " pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:29.105323 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-systemd-units\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.105323 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105260 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-cni-netd\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.105323 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105311 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-ovn-node-metrics-cert\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.105527 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105351 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 08 14:16:29.105527 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13b04066-34fd-477b-895e-fe98f76e661d-multus-daemon-config\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.105527 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:29.105527 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105467 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-socket-dir\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.105527 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105500 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-var-lib-openvswitch\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.105770 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105534 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-ovnkube-config\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.105770 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-cnibin\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.105770 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105599 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-var-lib-cni-multus\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.105770 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105648 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 08 14:16:29.105770 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105651 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-run-multus-certs\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.105770 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.106045 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105796 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-multus-cni-dir\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.106045 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105839 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-var-lib-kubelet\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.106045 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105868 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf954761-020d-4a65-8997-5b4e59b79748-iptables-alerter-script\") pod \"iptables-alerter-pxmln\" (UID: \"cf954761-020d-4a65-8997-5b4e59b79748\") " pod="openshift-network-operator/iptables-alerter-pxmln" Sep 08 14:16:29.106045 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105905 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e80f39e-3303-4f76-b8aa-bccfff9edff6-konnectivity-ca\") pod \"konnectivity-agent-tpxxk\" (UID: \"2e80f39e-3303-4f76-b8aa-bccfff9edff6\") " pod="kube-system/konnectivity-agent-tpxxk" Sep 08 14:16:29.106045 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105937 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4afcd3a-ef6b-4488-89b7-c48529821f6c-dbus\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:29.106045 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.105989 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwrpz\" (UniqueName: \"kubernetes.io/projected/5267f2dc-393e-40f2-8b78-25179961c1a3-kube-api-access-pwrpz\") pod \"network-metrics-daemon-998rf\" (UID: \"5267f2dc-393e-40f2-8b78-25179961c1a3\") " pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:29.106045 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.106021 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-registration-dir\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.106270 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.106047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-ovnkube-script-lib\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.107007 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.106984 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-system-cni-dir\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.107067 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107031 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-run-ovn\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.107067 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khsm5\" (UniqueName: \"kubernetes.io/projected/13b04066-34fd-477b-895e-fe98f76e661d-kube-api-access-khsm5\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.107165 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107086 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4afcd3a-ef6b-4488-89b7-c48529821f6c-kubelet-config\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:29.107165 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107110 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-multus-conf-dir\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.107165 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.106989 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jffcq\"" Sep 08 14:16:29.107165 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d5bf364-23e4-42d1-b576-92465114d3c8-cnibin\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.107165 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107157 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-device-dir\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.107411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-etc-openvswitch\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.107411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107217 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4d496bc-aa63-48dd-b346-e04b0468e03a-host\") pod \"node-ca-hlpbr\" (UID: \"a4d496bc-aa63-48dd-b346-e04b0468e03a\") " pod="openshift-image-registry/node-ca-hlpbr" Sep 08 14:16:29.107411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107233 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-run-k8s-cni-cncf-io\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.107411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-etc-kubernetes\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.107411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107271 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-log-socket\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.107411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a4d496bc-aa63-48dd-b346-e04b0468e03a-serviceca\") pod \"node-ca-hlpbr\" (UID: \"a4d496bc-aa63-48dd-b346-e04b0468e03a\") " pod="openshift-image-registry/node-ca-hlpbr" Sep 08 14:16:29.107411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107318 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5bq\" (UniqueName: \"kubernetes.io/projected/a4d496bc-aa63-48dd-b346-e04b0468e03a-kube-api-access-ft5bq\") pod \"node-ca-hlpbr\" (UID: \"a4d496bc-aa63-48dd-b346-e04b0468e03a\") " pod="openshift-image-registry/node-ca-hlpbr" Sep 08 14:16:29.107411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q24qx\" (UniqueName: \"kubernetes.io/projected/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-kube-api-access-q24qx\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.107411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107372 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-run-netns\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.107744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107430 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-hostroot\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.107744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107464 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf954761-020d-4a65-8997-5b4e59b79748-host-slash\") pod \"iptables-alerter-pxmln\" (UID: \"cf954761-020d-4a65-8997-5b4e59b79748\") " pod="openshift-network-operator/iptables-alerter-pxmln" Sep 08 14:16:29.107744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvcl\" (UniqueName: \"kubernetes.io/projected/cf954761-020d-4a65-8997-5b4e59b79748-kube-api-access-zlvcl\") pod \"iptables-alerter-pxmln\" (UID: \"cf954761-020d-4a65-8997-5b4e59b79748\") " pod="openshift-network-operator/iptables-alerter-pxmln" Sep 08 14:16:29.107744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107519 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d5bf364-23e4-42d1-b576-92465114d3c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.107744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs\") pod \"network-metrics-daemon-998rf\" (UID: \"5267f2dc-393e-40f2-8b78-25179961c1a3\") " pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:29.107744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.107744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107598 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d5bf364-23e4-42d1-b576-92465114d3c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.107744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107614 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-run-openvswitch\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.107744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-run-ovn-kubernetes\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.107744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107654 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-multus-socket-dir-parent\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.107744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107669 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.107744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107698 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-run-systemd\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.107744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13b04066-34fd-477b-895e-fe98f76e661d-cni-binary-copy\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.108335 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107769 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e80f39e-3303-4f76-b8aa-bccfff9edff6-agent-certs\") pod \"konnectivity-agent-tpxxk\" (UID: \"2e80f39e-3303-4f76-b8aa-bccfff9edff6\") " pod="kube-system/konnectivity-agent-tpxxk" Sep 08 14:16:29.108335 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d5bf364-23e4-42d1-b576-92465114d3c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.108335 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s48j\" (UniqueName: \"kubernetes.io/projected/9d5bf364-23e4-42d1-b576-92465114d3c8-kube-api-access-5s48j\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.108335 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107822 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-env-overrides\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.108335 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107843 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-var-lib-cni-bin\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.108335 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107870 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-kubelet\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.108335 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.107891 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-run-netns\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.110220 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.110201 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 08 14:16:29.114222 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.114129 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 14:11:28 +0000 UTC" deadline="2028-02-20 01:55:08.104060326 +0000 UTC" Sep 08 14:16:29.114279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.114227 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12707h38m38.989838977s" Sep 08 14:16:29.139861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.139842 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2zqvx" Sep 08 14:16:29.152940 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.152889 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2zqvx" Sep 08 14:16:29.199292 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.199271 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 08 14:16:29.208705 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208680 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-multus-conf-dir\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.208804 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208709 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-sysctl-d\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.208804 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d5bf364-23e4-42d1-b576-92465114d3c8-cnibin\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.208804 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-device-dir\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.208804 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-etc-openvswitch\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.208804 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4d496bc-aa63-48dd-b346-e04b0468e03a-host\") pod \"node-ca-hlpbr\" (UID: \"a4d496bc-aa63-48dd-b346-e04b0468e03a\") " pod="openshift-image-registry/node-ca-hlpbr" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208811 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-run-k8s-cni-cncf-io\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-etc-kubernetes\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208849 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-sysconfig\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-log-socket\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208869 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9d5bf364-23e4-42d1-b576-92465114d3c8-cnibin\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208901 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a4d496bc-aa63-48dd-b346-e04b0468e03a-serviceca\") pod \"node-ca-hlpbr\" (UID: \"a4d496bc-aa63-48dd-b346-e04b0468e03a\") " pod="openshift-image-registry/node-ca-hlpbr" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208906 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-run-k8s-cni-cncf-io\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208925 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-multus-conf-dir\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208904 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-device-dir\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208943 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-log-socket\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208892 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-etc-openvswitch\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208929 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-sysctl-conf\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208956 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-etc-kubernetes\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57r4q\" (UniqueName: \"kubernetes.io/projected/0a40edec-ceaa-46c6-8108-0d86cc93bad4-kube-api-access-57r4q\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.208932 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4d496bc-aa63-48dd-b346-e04b0468e03a-host\") pod \"node-ca-hlpbr\" (UID: \"a4d496bc-aa63-48dd-b346-e04b0468e03a\") " pod="openshift-image-registry/node-ca-hlpbr" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5bq\" (UniqueName: \"kubernetes.io/projected/a4d496bc-aa63-48dd-b346-e04b0468e03a-kube-api-access-ft5bq\") pod \"node-ca-hlpbr\" (UID: \"a4d496bc-aa63-48dd-b346-e04b0468e03a\") " pod="openshift-image-registry/node-ca-hlpbr" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q24qx\" (UniqueName: \"kubernetes.io/projected/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-kube-api-access-q24qx\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.209112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-run-netns\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209108 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-hostroot\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf954761-020d-4a65-8997-5b4e59b79748-host-slash\") pod \"iptables-alerter-pxmln\" (UID: \"cf954761-020d-4a65-8997-5b4e59b79748\") " pod="openshift-network-operator/iptables-alerter-pxmln" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvcl\" (UniqueName: \"kubernetes.io/projected/cf954761-020d-4a65-8997-5b4e59b79748-kube-api-access-zlvcl\") pod \"iptables-alerter-pxmln\" (UID: \"cf954761-020d-4a65-8997-5b4e59b79748\") " pod="openshift-network-operator/iptables-alerter-pxmln" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209158 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d5bf364-23e4-42d1-b576-92465114d3c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209175 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-run-netns\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209194 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-hostroot\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209193 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf954761-020d-4a65-8997-5b4e59b79748-host-slash\") pod \"iptables-alerter-pxmln\" (UID: \"cf954761-020d-4a65-8997-5b4e59b79748\") " pod="openshift-network-operator/iptables-alerter-pxmln" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209224 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs\") pod \"network-metrics-daemon-998rf\" (UID: \"5267f2dc-393e-40f2-8b78-25179961c1a3\") " pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209266 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv894\" (UniqueName: \"kubernetes.io/projected/f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c-kube-api-access-mv894\") pod \"node-resolver-xrx9d\" (UID: \"f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c\") " pod="openshift-dns/node-resolver-xrx9d" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d5bf364-23e4-42d1-b576-92465114d3c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-run-openvswitch\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209325 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-run-ovn-kubernetes\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.209357 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209376 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-run-openvswitch\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.210022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-multus-socket-dir-parent\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209403 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-multus-socket-dir-parent\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209442 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-run-ovn-kubernetes\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.209454 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs podName:5267f2dc-393e-40f2-8b78-25179961c1a3 nodeName:}" failed. No retries permitted until 2026-09-08 14:16:29.709410952 +0000 UTC m=+1.993728070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs") pod "network-metrics-daemon-998rf" (UID: "5267f2dc-393e-40f2-8b78-25179961c1a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209445 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9d5bf364-23e4-42d1-b576-92465114d3c8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209508 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-lib-modules\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-run-systemd\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13b04066-34fd-477b-895e-fe98f76e661d-cni-binary-copy\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209646 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e80f39e-3303-4f76-b8aa-bccfff9edff6-agent-certs\") pod \"konnectivity-agent-tpxxk\" (UID: \"2e80f39e-3303-4f76-b8aa-bccfff9edff6\") " pod="kube-system/konnectivity-agent-tpxxk" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209657 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-run-systemd\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209667 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-etc-selinux\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d5bf364-23e4-42d1-b576-92465114d3c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s48j\" (UniqueName: \"kubernetes.io/projected/9d5bf364-23e4-42d1-b576-92465114d3c8-kube-api-access-5s48j\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-env-overrides\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-var-lib-cni-bin\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c-hosts-file\") pod \"node-resolver-xrx9d\" (UID: \"f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c\") " pod="openshift-dns/node-resolver-xrx9d" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-kubelet\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209867 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-run-netns\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-sys-fs\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk4cc\" (UniqueName: \"kubernetes.io/projected/df4f4a97-b500-41f7-bc90-dca2520927f6-kube-api-access-jk4cc\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209934 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a4d496bc-aa63-48dd-b346-e04b0468e03a-serviceca\") pod \"node-ca-hlpbr\" (UID: \"a4d496bc-aa63-48dd-b346-e04b0468e03a\") " pod="openshift-image-registry/node-ca-hlpbr" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209950 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-slash\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-host\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.209957 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-node-log\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210045 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-var-lib-cni-bin\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210060 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-cni-bin\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-run\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a40edec-ceaa-46c6-8108-0d86cc93bad4-tmp\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210140 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5bf364-23e4-42d1-b576-92465114d3c8-system-cni-dir\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d5bf364-23e4-42d1-b576-92465114d3c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8d5w\" (UniqueName: \"kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w\") pod \"network-check-target-4njj8\" (UID: \"43ad5ec2-69c5-4c3a-a997-adb20472335a\") " pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:29.211840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-systemd-units\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210240 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9d5bf364-23e4-42d1-b576-92465114d3c8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-cni-netd\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210276 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-ovn-node-metrics-cert\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210278 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13b04066-34fd-477b-895e-fe98f76e661d-cni-binary-copy\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210301 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-kubelet\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210281 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9d5bf364-23e4-42d1-b576-92465114d3c8-cni-binary-copy\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210303 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13b04066-34fd-477b-895e-fe98f76e661d-multus-daemon-config\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210345 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-node-log\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210356 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-run-netns\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210459 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9d5bf364-23e4-42d1-b576-92465114d3c8-system-cni-dir\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210465 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-slash\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210510 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-sys-fs\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210511 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-cni-netd\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-var-lib-kubelet\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210597 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-socket-dir\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.212560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-var-lib-openvswitch\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-ovnkube-config\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.210671 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-cnibin\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210716 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-var-lib-cni-multus\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.210731 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret podName:b4afcd3a-ef6b-4488-89b7-c48529821f6c nodeName:}" failed. No retries permitted until 2026-09-08 14:16:29.710714682 +0000 UTC m=+1.995031778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret") pod "global-pull-secret-syncer-hsh78" (UID: "b4afcd3a-ef6b-4488-89b7-c48529821f6c") : object "kube-system"/"original-pull-secret" not registered Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210775 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-env-overrides\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210834 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-var-lib-cni-multus\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210894 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/13b04066-34fd-477b-895e-fe98f76e661d-multus-daemon-config\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210903 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9d5bf364-23e4-42d1-b576-92465114d3c8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210894 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-run-multus-certs\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210931 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-run-multus-certs\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210939 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210971 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-host-cni-bin\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.210977 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-cnibin\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211027 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-multus-cni-dir\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211053 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-var-lib-kubelet\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.213080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211082 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-modprobe-d\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211098 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-multus-cni-dir\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211097 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-var-lib-openvswitch\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211145 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-sys\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-tuned\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211188 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-socket-dir\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211205 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-systemd-units\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211172 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211205 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf954761-020d-4a65-8997-5b4e59b79748-iptables-alerter-script\") pod \"iptables-alerter-pxmln\" (UID: \"cf954761-020d-4a65-8997-5b4e59b79748\") " pod="openshift-network-operator/iptables-alerter-pxmln" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211257 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e80f39e-3303-4f76-b8aa-bccfff9edff6-konnectivity-ca\") pod \"konnectivity-agent-tpxxk\" (UID: \"2e80f39e-3303-4f76-b8aa-bccfff9edff6\") " pod="kube-system/konnectivity-agent-tpxxk" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211258 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-host-var-lib-kubelet\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211285 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-ovnkube-config\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211343 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4afcd3a-ef6b-4488-89b7-c48529821f6c-dbus\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211470 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwrpz\" (UniqueName: \"kubernetes.io/projected/5267f2dc-393e-40f2-8b78-25179961c1a3-kube-api-access-pwrpz\") pod \"network-metrics-daemon-998rf\" (UID: \"5267f2dc-393e-40f2-8b78-25179961c1a3\") " pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-registration-dir\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-ovnkube-script-lib\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211622 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-system-cni-dir\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.213572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211656 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-kubernetes\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211607 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b4afcd3a-ef6b-4488-89b7-c48529821f6c-dbus\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-run-ovn\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211720 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/df4f4a97-b500-41f7-bc90-dca2520927f6-registration-dir\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211728 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf954761-020d-4a65-8997-5b4e59b79748-iptables-alerter-script\") pod \"iptables-alerter-pxmln\" (UID: \"cf954761-020d-4a65-8997-5b4e59b79748\") " pod="openshift-network-operator/iptables-alerter-pxmln" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khsm5\" (UniqueName: \"kubernetes.io/projected/13b04066-34fd-477b-895e-fe98f76e661d-kube-api-access-khsm5\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211779 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-run-ovn\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211815 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-systemd\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211845 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13b04066-34fd-477b-895e-fe98f76e661d-system-cni-dir\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211885 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c-tmp-dir\") pod \"node-resolver-xrx9d\" (UID: \"f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c\") " pod="openshift-dns/node-resolver-xrx9d" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.211916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4afcd3a-ef6b-4488-89b7-c48529821f6c-kubelet-config\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.212016 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b4afcd3a-ef6b-4488-89b7-c48529821f6c-kubelet-config\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.212106 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2e80f39e-3303-4f76-b8aa-bccfff9edff6-konnectivity-ca\") pod \"konnectivity-agent-tpxxk\" (UID: \"2e80f39e-3303-4f76-b8aa-bccfff9edff6\") " pod="kube-system/konnectivity-agent-tpxxk" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.212223 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-ovnkube-script-lib\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.214091 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.213998 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-ovn-node-metrics-cert\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.214884 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.214859 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2e80f39e-3303-4f76-b8aa-bccfff9edff6-agent-certs\") pod \"konnectivity-agent-tpxxk\" (UID: \"2e80f39e-3303-4f76-b8aa-bccfff9edff6\") " pod="kube-system/konnectivity-agent-tpxxk" Sep 08 14:16:29.222082 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.222065 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5bq\" (UniqueName: \"kubernetes.io/projected/a4d496bc-aa63-48dd-b346-e04b0468e03a-kube-api-access-ft5bq\") pod \"node-ca-hlpbr\" (UID: \"a4d496bc-aa63-48dd-b346-e04b0468e03a\") " pod="openshift-image-registry/node-ca-hlpbr" Sep 08 14:16:29.226064 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.224692 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 14:16:29.226064 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.224720 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 14:16:29.226064 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.224737 2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8d5w for pod openshift-network-diagnostics/network-check-target-4njj8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:16:29.226064 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.224827 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w podName:43ad5ec2-69c5-4c3a-a997-adb20472335a nodeName:}" failed. No retries permitted until 2026-09-08 14:16:29.724785345 +0000 UTC m=+2.009102440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-f8d5w" (UniqueName: "kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w") pod "network-check-target-4njj8" (UID: "43ad5ec2-69c5-4c3a-a997-adb20472335a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:16:29.227076 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.227041 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q24qx\" (UniqueName: \"kubernetes.io/projected/1a3ac79a-e3b0-4b49-aa08-d77e1d12e397-kube-api-access-q24qx\") pod \"ovnkube-node-tlbbx\" (UID: \"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397\") " pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.227906 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.227886 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s48j\" (UniqueName: \"kubernetes.io/projected/9d5bf364-23e4-42d1-b576-92465114d3c8-kube-api-access-5s48j\") pod \"multus-additional-cni-plugins-6kt9z\" (UID: \"9d5bf364-23e4-42d1-b576-92465114d3c8\") " pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.228638 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.228615 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwrpz\" (UniqueName: \"kubernetes.io/projected/5267f2dc-393e-40f2-8b78-25179961c1a3-kube-api-access-pwrpz\") pod \"network-metrics-daemon-998rf\" (UID: \"5267f2dc-393e-40f2-8b78-25179961c1a3\") " pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:29.228719 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.228619 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvcl\" (UniqueName: \"kubernetes.io/projected/cf954761-020d-4a65-8997-5b4e59b79748-kube-api-access-zlvcl\") pod \"iptables-alerter-pxmln\" (UID: \"cf954761-020d-4a65-8997-5b4e59b79748\") " pod="openshift-network-operator/iptables-alerter-pxmln" Sep 08 14:16:29.229627 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.229606 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk4cc\" (UniqueName: \"kubernetes.io/projected/df4f4a97-b500-41f7-bc90-dca2520927f6-kube-api-access-jk4cc\") pod \"aws-ebs-csi-driver-node-sgfg2\" (UID: \"df4f4a97-b500-41f7-bc90-dca2520927f6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.229700 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.229609 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khsm5\" (UniqueName: \"kubernetes.io/projected/13b04066-34fd-477b-895e-fe98f76e661d-kube-api-access-khsm5\") pod \"multus-wrvcw\" (UID: \"13b04066-34fd-477b-895e-fe98f76e661d\") " pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.312774 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.312736 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-var-lib-kubelet\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.312934 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.312793 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-modprobe-d\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.312934 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.312816 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-sys\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.312934 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.312877 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-var-lib-kubelet\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313104 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.312971 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-modprobe-d\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313104 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.312972 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-sys\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313104 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-tuned\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313104 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-kubernetes\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313104 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-systemd\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313365 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313106 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-kubernetes\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313365 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313110 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c-tmp-dir\") pod \"node-resolver-xrx9d\" (UID: \"f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c\") " pod="openshift-dns/node-resolver-xrx9d" Sep 08 14:16:29.313365 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-sysctl-d\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313365 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313143 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-systemd\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313365 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-sysconfig\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313365 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-sysctl-conf\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313365 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313240 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-sysctl-d\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313365 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57r4q\" (UniqueName: \"kubernetes.io/projected/0a40edec-ceaa-46c6-8108-0d86cc93bad4-kube-api-access-57r4q\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313365 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313303 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-sysconfig\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313365 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv894\" (UniqueName: \"kubernetes.io/projected/f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c-kube-api-access-mv894\") pod \"node-resolver-xrx9d\" (UID: \"f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c\") " pod="openshift-dns/node-resolver-xrx9d" Sep 08 14:16:29.313365 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-lib-modules\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313941 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313385 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c-tmp-dir\") pod \"node-resolver-xrx9d\" (UID: \"f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c\") " pod="openshift-dns/node-resolver-xrx9d" Sep 08 14:16:29.313941 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313396 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-sysctl-conf\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313941 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c-hosts-file\") pod \"node-resolver-xrx9d\" (UID: \"f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c\") " pod="openshift-dns/node-resolver-xrx9d" Sep 08 14:16:29.313941 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-host\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313941 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-run\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313941 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313509 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-host\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313941 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a40edec-ceaa-46c6-8108-0d86cc93bad4-tmp\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313941 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313544 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-lib-modules\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313941 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313641 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0a40edec-ceaa-46c6-8108-0d86cc93bad4-run\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.313941 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.313743 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c-hosts-file\") pod \"node-resolver-xrx9d\" (UID: \"f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c\") " pod="openshift-dns/node-resolver-xrx9d" Sep 08 14:16:29.315778 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.315761 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0a40edec-ceaa-46c6-8108-0d86cc93bad4-tmp\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.316240 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.316220 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0a40edec-ceaa-46c6-8108-0d86cc93bad4-etc-tuned\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.325513 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.325485 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv894\" (UniqueName: \"kubernetes.io/projected/f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c-kube-api-access-mv894\") pod \"node-resolver-xrx9d\" (UID: \"f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c\") " pod="openshift-dns/node-resolver-xrx9d" Sep 08 14:16:29.325599 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.325552 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57r4q\" (UniqueName: \"kubernetes.io/projected/0a40edec-ceaa-46c6-8108-0d86cc93bad4-kube-api-access-57r4q\") pod \"tuned-vnl5t\" (UID: \"0a40edec-ceaa-46c6-8108-0d86cc93bad4\") " pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.413146 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.413083 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:29.430500 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:29.430456 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3ac79a_e3b0_4b49_aa08_d77e1d12e397.slice/crio-bfd40365691e24132c325c6e1f7fa47fe5cb02d324633df12a2d5d585706c233 WatchSource:0}: Error finding container bfd40365691e24132c325c6e1f7fa47fe5cb02d324633df12a2d5d585706c233: Status 404 returned error can't find the container with id bfd40365691e24132c325c6e1f7fa47fe5cb02d324633df12a2d5d585706c233 Sep 08 14:16:29.431082 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:29.431062 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40496547e3481087fc874b11f52284a9.slice/crio-baddddcf8cbfb57ccde1618b172d0eac6ddcbf2ecb32deb11925b85e9d6e8037 WatchSource:0}: Error finding container baddddcf8cbfb57ccde1618b172d0eac6ddcbf2ecb32deb11925b85e9d6e8037: Status 404 returned error can't find the container with id baddddcf8cbfb57ccde1618b172d0eac6ddcbf2ecb32deb11925b85e9d6e8037 Sep 08 14:16:29.431701 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:29.431676 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308f44f9483500741181dba63fa91da4.slice/crio-59fc796247f6f419e07ea3bda176513761cf69708af9ccd5bdaced8514471610 WatchSource:0}: Error finding container 59fc796247f6f419e07ea3bda176513761cf69708af9ccd5bdaced8514471610: Status 404 returned error can't find the container with id 59fc796247f6f419e07ea3bda176513761cf69708af9ccd5bdaced8514471610 Sep 08 14:16:29.435958 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.435934 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hlpbr" Sep 08 14:16:29.436067 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.436007 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 14:16:29.443963 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:29.443941 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d496bc_aa63_48dd_b346_e04b0468e03a.slice/crio-b86f06d1fa28bf34536ac60d8e27ab6adcabd833a1ea9c12f18829cbedc82734 WatchSource:0}: Error finding container b86f06d1fa28bf34536ac60d8e27ab6adcabd833a1ea9c12f18829cbedc82734: Status 404 returned error can't find the container with id b86f06d1fa28bf34536ac60d8e27ab6adcabd833a1ea9c12f18829cbedc82734 Sep 08 14:16:29.444529 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.444510 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 14:16:29.445457 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.445443 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wrvcw" Sep 08 14:16:29.453397 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:29.453378 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13b04066_34fd_477b_895e_fe98f76e661d.slice/crio-d1deac19c70af598cd85c8c4f49f7d1bbc00ee3a258593c175d34c838e0d0308 WatchSource:0}: Error finding container d1deac19c70af598cd85c8c4f49f7d1bbc00ee3a258593c175d34c838e0d0308: Status 404 returned error can't find the container with id d1deac19c70af598cd85c8c4f49f7d1bbc00ee3a258593c175d34c838e0d0308 Sep 08 14:16:29.471787 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.471763 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pxmln" Sep 08 14:16:29.478349 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:29.478329 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf954761_020d_4a65_8997_5b4e59b79748.slice/crio-35420c33219f0fa6a93884fc41c7d951c91a62b8e9cce76b7ee36148b42ebbf5 WatchSource:0}: Error finding container 35420c33219f0fa6a93884fc41c7d951c91a62b8e9cce76b7ee36148b42ebbf5: Status 404 returned error can't find the container with id 35420c33219f0fa6a93884fc41c7d951c91a62b8e9cce76b7ee36148b42ebbf5 Sep 08 14:16:29.493136 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.493114 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tpxxk" Sep 08 14:16:29.501323 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.501301 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6kt9z" Sep 08 14:16:29.502611 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:29.502542 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e80f39e_3303_4f76_b8aa_bccfff9edff6.slice/crio-b9c38b16e7f15577b7532be652428db684df2f3dd944f0e5116e783a0e8f3359 WatchSource:0}: Error finding container b9c38b16e7f15577b7532be652428db684df2f3dd944f0e5116e783a0e8f3359: Status 404 returned error can't find the container with id b9c38b16e7f15577b7532be652428db684df2f3dd944f0e5116e783a0e8f3359 Sep 08 14:16:29.508918 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:29.508895 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5bf364_23e4_42d1_b576_92465114d3c8.slice/crio-e37cd97f672d6c4f4ba13ecc27a3a99912f3071c78c99832a54328508741ab15 WatchSource:0}: Error finding container e37cd97f672d6c4f4ba13ecc27a3a99912f3071c78c99832a54328508741ab15: Status 404 returned error can't find the container with id e37cd97f672d6c4f4ba13ecc27a3a99912f3071c78c99832a54328508741ab15 Sep 08 14:16:29.513619 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.513576 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" Sep 08 14:16:29.520761 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:29.520743 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4f4a97_b500_41f7_bc90_dca2520927f6.slice/crio-1872e48b947382254bbd52fe0563d3388c76bbefeae02f525be9053e43026c00 WatchSource:0}: Error finding container 1872e48b947382254bbd52fe0563d3388c76bbefeae02f525be9053e43026c00: Status 404 returned error can't find the container with id 1872e48b947382254bbd52fe0563d3388c76bbefeae02f525be9053e43026c00 Sep 08 14:16:29.543629 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.543605 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" Sep 08 14:16:29.547823 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.547792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xrx9d" Sep 08 14:16:29.550751 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:29.550731 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a40edec_ceaa_46c6_8108_0d86cc93bad4.slice/crio-a667476bf541492e1b4853a16204786565774b567956a42028ccf7353c0b8e07 WatchSource:0}: Error finding container a667476bf541492e1b4853a16204786565774b567956a42028ccf7353c0b8e07: Status 404 returned error can't find the container with id a667476bf541492e1b4853a16204786565774b567956a42028ccf7353c0b8e07 Sep 08 14:16:29.555386 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:16:29.555368 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45cf07b_5cde_44a3_b0e0_2bc6bad77f7c.slice/crio-e7a2c8ed18f79ecd73380519d7a0400ff2835b9a6452287922ad5dfcad8202a2 WatchSource:0}: Error finding container e7a2c8ed18f79ecd73380519d7a0400ff2835b9a6452287922ad5dfcad8202a2: Status 404 returned error can't find the container with id e7a2c8ed18f79ecd73380519d7a0400ff2835b9a6452287922ad5dfcad8202a2 Sep 08 14:16:29.556950 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.556930 2578 kuberuntime_manager.go:1459] "Unhandled Error" err="container dns-node-resolver start failed in pod node-resolver-xrx9d_openshift-dns(f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Sep 08 14:16:29.557003 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.556966 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-dns/node-resolver-xrx9d" podUID="f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c" Sep 08 14:16:29.716857 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.716776 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs\") pod \"network-metrics-daemon-998rf\" (UID: \"5267f2dc-393e-40f2-8b78-25179961c1a3\") " pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:29.716857 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.716835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:29.717038 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.716890 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:16:29.717038 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.716926 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 14:16:29.717038 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.716951 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs podName:5267f2dc-393e-40f2-8b78-25179961c1a3 nodeName:}" failed. No retries permitted until 2026-09-08 14:16:30.716934212 +0000 UTC m=+3.001251307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs") pod "network-metrics-daemon-998rf" (UID: "5267f2dc-393e-40f2-8b78-25179961c1a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:16:29.717038 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.716968 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret podName:b4afcd3a-ef6b-4488-89b7-c48529821f6c nodeName:}" failed. No retries permitted until 2026-09-08 14:16:30.716957661 +0000 UTC m=+3.001274760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret") pod "global-pull-secret-syncer-hsh78" (UID: "b4afcd3a-ef6b-4488-89b7-c48529821f6c") : object "kube-system"/"original-pull-secret" not registered Sep 08 14:16:29.817665 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:29.817636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8d5w\" (UniqueName: \"kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w\") pod \"network-check-target-4njj8\" (UID: \"43ad5ec2-69c5-4c3a-a997-adb20472335a\") " pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:29.817842 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.817815 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 14:16:29.817842 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.817835 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 14:16:29.817915 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.817845 2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8d5w for pod openshift-network-diagnostics/network-check-target-4njj8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:16:29.817915 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:29.817894 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w podName:43ad5ec2-69c5-4c3a-a997-adb20472335a nodeName:}" failed. No retries permitted until 2026-09-08 14:16:30.817881011 +0000 UTC m=+3.102198105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8d5w" (UniqueName: "kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w") pod "network-check-target-4njj8" (UID: "43ad5ec2-69c5-4c3a-a997-adb20472335a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:16:30.155103 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.154980 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 14:11:29 +0000 UTC" deadline="2028-06-22 16:47:55.832994798 +0000 UTC" Sep 08 14:16:30.155103 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.155049 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15674h31m25.677950322s" Sep 08 14:16:30.238575 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.238541 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:30.238760 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:30.238692 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:30.238859 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.238779 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:30.238923 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:30.238876 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:30.269623 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.269561 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xrx9d" event={"ID":"f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c","Type":"ContainerStarted","Data":"e7a2c8ed18f79ecd73380519d7a0400ff2835b9a6452287922ad5dfcad8202a2"} Sep 08 14:16:30.274170 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:30.274132 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"dns-node-resolver\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40bc1b9c6b608369978e9f165c5e22d7a75fd91b3480cba60cd275228d721616\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-dns/node-resolver-xrx9d" podUID="f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c" Sep 08 14:16:30.293953 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.293830 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" event={"ID":"0a40edec-ceaa-46c6-8108-0d86cc93bad4","Type":"ContainerStarted","Data":"a667476bf541492e1b4853a16204786565774b567956a42028ccf7353c0b8e07"} Sep 08 14:16:30.299685 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.299604 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kt9z" event={"ID":"9d5bf364-23e4-42d1-b576-92465114d3c8","Type":"ContainerStarted","Data":"e37cd97f672d6c4f4ba13ecc27a3a99912f3071c78c99832a54328508741ab15"} Sep 08 14:16:30.301432 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.301315 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tpxxk" event={"ID":"2e80f39e-3303-4f76-b8aa-bccfff9edff6","Type":"ContainerStarted","Data":"b9c38b16e7f15577b7532be652428db684df2f3dd944f0e5116e783a0e8f3359"} Sep 08 14:16:30.303904 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.303838 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal" event={"ID":"308f44f9483500741181dba63fa91da4","Type":"ContainerStarted","Data":"59fc796247f6f419e07ea3bda176513761cf69708af9ccd5bdaced8514471610"} Sep 08 14:16:30.306601 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.306574 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" event={"ID":"df4f4a97-b500-41f7-bc90-dca2520927f6","Type":"ContainerStarted","Data":"1872e48b947382254bbd52fe0563d3388c76bbefeae02f525be9053e43026c00"} Sep 08 14:16:30.312170 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.312142 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pxmln" event={"ID":"cf954761-020d-4a65-8997-5b4e59b79748","Type":"ContainerStarted","Data":"35420c33219f0fa6a93884fc41c7d951c91a62b8e9cce76b7ee36148b42ebbf5"} Sep 08 14:16:30.341019 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.340976 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wrvcw" event={"ID":"13b04066-34fd-477b-895e-fe98f76e661d","Type":"ContainerStarted","Data":"d1deac19c70af598cd85c8c4f49f7d1bbc00ee3a258593c175d34c838e0d0308"} Sep 08 14:16:30.354533 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.354383 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hlpbr" event={"ID":"a4d496bc-aa63-48dd-b346-e04b0468e03a","Type":"ContainerStarted","Data":"b86f06d1fa28bf34536ac60d8e27ab6adcabd833a1ea9c12f18829cbedc82734"} Sep 08 14:16:30.376932 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.376896 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" event={"ID":"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397","Type":"ContainerStarted","Data":"bfd40365691e24132c325c6e1f7fa47fe5cb02d324633df12a2d5d585706c233"} Sep 08 14:16:30.394509 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.394465 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" event={"ID":"40496547e3481087fc874b11f52284a9","Type":"ContainerStarted","Data":"baddddcf8cbfb57ccde1618b172d0eac6ddcbf2ecb32deb11925b85e9d6e8037"} Sep 08 14:16:30.527492 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.527167 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 14:16:30.724327 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.724287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:30.724540 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.724372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs\") pod \"network-metrics-daemon-998rf\" (UID: \"5267f2dc-393e-40f2-8b78-25179961c1a3\") " pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:30.724540 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:30.724516 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:16:30.724683 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:30.724587 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs podName:5267f2dc-393e-40f2-8b78-25179961c1a3 nodeName:}" failed. No retries permitted until 2026-09-08 14:16:32.72456658 +0000 UTC m=+5.008883687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs") pod "network-metrics-daemon-998rf" (UID: "5267f2dc-393e-40f2-8b78-25179961c1a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:16:30.725079 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:30.725054 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 14:16:30.725186 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:30.725117 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret podName:b4afcd3a-ef6b-4488-89b7-c48529821f6c nodeName:}" failed. No retries permitted until 2026-09-08 14:16:32.725099461 +0000 UTC m=+5.009416566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret") pod "global-pull-secret-syncer-hsh78" (UID: "b4afcd3a-ef6b-4488-89b7-c48529821f6c") : object "kube-system"/"original-pull-secret" not registered Sep 08 14:16:30.825182 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.825094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8d5w\" (UniqueName: \"kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w\") pod \"network-check-target-4njj8\" (UID: \"43ad5ec2-69c5-4c3a-a997-adb20472335a\") " pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:30.825341 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:30.825281 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 14:16:30.825341 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:30.825301 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 14:16:30.825341 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:30.825316 2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8d5w for pod openshift-network-diagnostics/network-check-target-4njj8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:16:30.825496 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:30.825375 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w podName:43ad5ec2-69c5-4c3a-a997-adb20472335a nodeName:}" failed. No retries permitted until 2026-09-08 14:16:32.825356195 +0000 UTC m=+5.109673311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8d5w" (UniqueName: "kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w") pod "network-check-target-4njj8" (UID: "43ad5ec2-69c5-4c3a-a997-adb20472335a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:16:30.857421 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:30.857390 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 08 14:16:31.155895 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:31.155804 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 14:11:29 +0000 UTC" deadline="2028-04-23 17:27:05.907499479 +0000 UTC" Sep 08 14:16:31.155895 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:31.155845 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14235h10m34.751659457s" Sep 08 14:16:31.234996 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:31.234967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:31.235160 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:31.235091 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:32.235672 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:32.235235 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:32.235672 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:32.235396 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:32.235672 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:32.235444 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:32.235672 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:32.235549 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:32.742236 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:32.742188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:32.742385 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:32.742318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs\") pod \"network-metrics-daemon-998rf\" (UID: \"5267f2dc-393e-40f2-8b78-25179961c1a3\") " pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:32.742495 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:32.742476 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:16:32.742597 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:32.742555 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs podName:5267f2dc-393e-40f2-8b78-25179961c1a3 nodeName:}" failed. No retries permitted until 2026-09-08 14:16:36.742528546 +0000 UTC m=+9.026845644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs") pod "network-metrics-daemon-998rf" (UID: "5267f2dc-393e-40f2-8b78-25179961c1a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:16:32.742871 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:32.742847 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 14:16:32.742955 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:32.742903 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret podName:b4afcd3a-ef6b-4488-89b7-c48529821f6c nodeName:}" failed. No retries permitted until 2026-09-08 14:16:36.742887518 +0000 UTC m=+9.027204627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret") pod "global-pull-secret-syncer-hsh78" (UID: "b4afcd3a-ef6b-4488-89b7-c48529821f6c") : object "kube-system"/"original-pull-secret" not registered Sep 08 14:16:32.842863 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:32.842826 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8d5w\" (UniqueName: \"kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w\") pod \"network-check-target-4njj8\" (UID: \"43ad5ec2-69c5-4c3a-a997-adb20472335a\") " pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:32.843052 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:32.843002 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 14:16:32.843052 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:32.843018 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 14:16:32.843052 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:32.843027 2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8d5w for pod openshift-network-diagnostics/network-check-target-4njj8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:16:32.843254 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:32.843083 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w podName:43ad5ec2-69c5-4c3a-a997-adb20472335a nodeName:}" failed. No retries permitted until 2026-09-08 14:16:36.843068384 +0000 UTC m=+9.127385483 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8d5w" (UniqueName: "kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w") pod "network-check-target-4njj8" (UID: "43ad5ec2-69c5-4c3a-a997-adb20472335a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:16:33.235897 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:33.235814 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:33.236374 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:33.235951 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:34.236179 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:34.236144 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:34.236636 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:34.236192 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:34.236636 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:34.236276 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:34.236636 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:34.236419 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:35.236438 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:35.235999 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:35.236438 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:35.236125 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:36.235541 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:36.235508 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:36.235541 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:36.235533 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:36.235790 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:36.235646 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:36.235859 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:36.235794 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:36.775993 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:36.775949 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs\") pod \"network-metrics-daemon-998rf\" (UID: \"5267f2dc-393e-40f2-8b78-25179961c1a3\") " pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:36.776430 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:36.776031 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:36.776430 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:36.776124 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:16:36.776430 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:36.776151 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 14:16:36.776430 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:36.776200 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs podName:5267f2dc-393e-40f2-8b78-25179961c1a3 nodeName:}" failed. No retries permitted until 2026-09-08 14:16:44.776179056 +0000 UTC m=+17.060496161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs") pod "network-metrics-daemon-998rf" (UID: "5267f2dc-393e-40f2-8b78-25179961c1a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:16:36.776430 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:36.776221 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret podName:b4afcd3a-ef6b-4488-89b7-c48529821f6c nodeName:}" failed. No retries permitted until 2026-09-08 14:16:44.776211106 +0000 UTC m=+17.060528205 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret") pod "global-pull-secret-syncer-hsh78" (UID: "b4afcd3a-ef6b-4488-89b7-c48529821f6c") : object "kube-system"/"original-pull-secret" not registered Sep 08 14:16:36.877040 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:36.876998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8d5w\" (UniqueName: \"kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w\") pod \"network-check-target-4njj8\" (UID: \"43ad5ec2-69c5-4c3a-a997-adb20472335a\") " pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:36.877201 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:36.877151 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 14:16:36.877201 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:36.877173 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 14:16:36.877201 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:36.877182 2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8d5w for pod openshift-network-diagnostics/network-check-target-4njj8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:16:36.877303 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:36.877237 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w podName:43ad5ec2-69c5-4c3a-a997-adb20472335a nodeName:}" failed. No retries permitted until 2026-09-08 14:16:44.877224085 +0000 UTC m=+17.161541178 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8d5w" (UniqueName: "kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w") pod "network-check-target-4njj8" (UID: "43ad5ec2-69c5-4c3a-a997-adb20472335a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:16:37.236218 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:37.235894 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:37.236218 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:37.236032 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:38.238023 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:38.237983 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:38.238425 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:38.238179 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:38.238425 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:38.238178 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:38.238425 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:38.238309 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:39.235351 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:39.235270 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:39.235514 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:39.235406 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:40.235557 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:40.235514 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:40.235557 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:40.235549 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:40.236087 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:40.235662 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:40.236087 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:40.235805 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:41.235987 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:41.235904 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:41.236343 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:41.236020 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:42.236125 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:42.236087 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:42.236595 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:42.236136 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:42.236595 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:42.236238 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:42.236595 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:42.236367 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:43.235991 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:43.235953 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:43.236160 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:43.236076 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:44.235581 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:44.235542 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:44.235780 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:44.235542 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:44.235780 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:44.235662 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:44.235895 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:44.235780 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:44.835544 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:44.835460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:44.835544 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:44.835527 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs\") pod \"network-metrics-daemon-998rf\" (UID: \"5267f2dc-393e-40f2-8b78-25179961c1a3\") " pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:44.836040 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:44.835625 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:16:44.836040 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:44.835625 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 14:16:44.836040 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:44.835700 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs podName:5267f2dc-393e-40f2-8b78-25179961c1a3 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:00.835679788 +0000 UTC m=+33.119996896 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs") pod "network-metrics-daemon-998rf" (UID: "5267f2dc-393e-40f2-8b78-25179961c1a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:16:44.836040 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:44.835723 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret podName:b4afcd3a-ef6b-4488-89b7-c48529821f6c nodeName:}" failed. No retries permitted until 2026-09-08 14:17:00.835710718 +0000 UTC m=+33.120027828 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret") pod "global-pull-secret-syncer-hsh78" (UID: "b4afcd3a-ef6b-4488-89b7-c48529821f6c") : object "kube-system"/"original-pull-secret" not registered Sep 08 14:16:44.936324 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:44.936289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8d5w\" (UniqueName: \"kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w\") pod \"network-check-target-4njj8\" (UID: \"43ad5ec2-69c5-4c3a-a997-adb20472335a\") " pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:44.936503 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:44.936459 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 14:16:44.936503 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:44.936486 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 14:16:44.936503 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:44.936495 2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8d5w for pod openshift-network-diagnostics/network-check-target-4njj8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:16:44.936644 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:44.936548 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w podName:43ad5ec2-69c5-4c3a-a997-adb20472335a nodeName:}" failed. No retries permitted until 2026-09-08 14:17:00.936532995 +0000 UTC m=+33.220850108 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8d5w" (UniqueName: "kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w") pod "network-check-target-4njj8" (UID: "43ad5ec2-69c5-4c3a-a997-adb20472335a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:16:45.236092 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:45.236013 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:45.236238 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:45.236129 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:46.235746 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:46.235697 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:46.236200 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:46.235697 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:46.236200 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:46.235856 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:46.236200 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:46.235899 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:47.235615 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:47.235574 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:47.235808 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:47.235718 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:48.236225 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:48.236193 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:48.236659 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:48.236311 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:48.236659 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:48.236373 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:48.236659 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:48.236482 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:49.236948 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.236302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:49.236948 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:49.236780 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:49.443068 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.443035 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" event={"ID":"0a40edec-ceaa-46c6-8108-0d86cc93bad4","Type":"ContainerStarted","Data":"de4595799f42f08ef8486d0ecc7b211a4ff342f08b7374913a92490ec60da121"} Sep 08 14:16:49.444754 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.444709 2578 generic.go:356] "Generic (PLEG): container finished" podID="9d5bf364-23e4-42d1-b576-92465114d3c8" containerID="6eff941a153e956a2b19eb8e9b7749f019ecf2b94002cc563383860f07454a73" exitCode=0 Sep 08 14:16:49.444862 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.444767 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kt9z" event={"ID":"9d5bf364-23e4-42d1-b576-92465114d3c8","Type":"ContainerDied","Data":"6eff941a153e956a2b19eb8e9b7749f019ecf2b94002cc563383860f07454a73"} Sep 08 14:16:49.446459 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.446433 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tpxxk" event={"ID":"2e80f39e-3303-4f76-b8aa-bccfff9edff6","Type":"ContainerStarted","Data":"7605fcbb898b7d4e7902e15cc5266c27fd95e98040f0398236682565dfb055c0"} Sep 08 14:16:49.448042 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.448013 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal" event={"ID":"308f44f9483500741181dba63fa91da4","Type":"ContainerStarted","Data":"4fbe56b41e7c7f2df45de5bfc7de88feb712ac2f557d26ccce3f64fe8b2e4546"} Sep 08 14:16:49.449632 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.449605 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" event={"ID":"df4f4a97-b500-41f7-bc90-dca2520927f6","Type":"ContainerStarted","Data":"d8fb30de11a212330f343e16d3905db68e02861c73b4ac1a977d936a0660abf1"} Sep 08 14:16:49.450968 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.450948 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wrvcw" event={"ID":"13b04066-34fd-477b-895e-fe98f76e661d","Type":"ContainerStarted","Data":"a9747d81cf42bd3ec43583ec9463a58d326cfb26848d371bb8c1cd0d8da17e4b"} Sep 08 14:16:49.452531 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.452514 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hlpbr" event={"ID":"a4d496bc-aa63-48dd-b346-e04b0468e03a","Type":"ContainerStarted","Data":"f6c70e3235ae90fc2a0e6c056057762203316ae62797ab25afd2d47432e38677"} Sep 08 14:16:49.455363 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.455342 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:16:49.455722 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.455695 2578 generic.go:356] "Generic (PLEG): container finished" podID="1a3ac79a-e3b0-4b49-aa08-d77e1d12e397" containerID="132ddbd85d9e99ffd3f2e0678de8c41a2a11ed651c89eb1173c43db49417c670" exitCode=1 Sep 08 14:16:49.455840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.455760 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" event={"ID":"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397","Type":"ContainerStarted","Data":"393a8e7cc6d1670c6e2c7cba05b7c24828a316ebdcb555bb653ef6cf07a3fd10"} Sep 08 14:16:49.455840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.455815 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" event={"ID":"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397","Type":"ContainerStarted","Data":"22b5f8d1793149d2d7a001b750da32f4efbfa1c9344197a9e9cb172981e8d244"} Sep 08 14:16:49.455940 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.455854 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" event={"ID":"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397","Type":"ContainerStarted","Data":"4135a0cf618bc34bbc63483545fc2dab0fa7005895a3cc02ef1fe2f9c403b957"} Sep 08 14:16:49.455940 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.455891 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" event={"ID":"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397","Type":"ContainerStarted","Data":"e19ed1f5a1cdfc50e0d3ddfd8faf19633d783a2fac49ad65cbf736347d46f64a"} Sep 08 14:16:49.455940 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.455934 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" event={"ID":"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397","Type":"ContainerDied","Data":"132ddbd85d9e99ffd3f2e0678de8c41a2a11ed651c89eb1173c43db49417c670"} Sep 08 14:16:49.456057 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.455963 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" event={"ID":"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397","Type":"ContainerStarted","Data":"050557b93c42725f05bb7bcea764e9c13cb944092db3e7c4da9af026ca26434c"} Sep 08 14:16:49.457340 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.457321 2578 generic.go:356] "Generic (PLEG): container finished" podID="40496547e3481087fc874b11f52284a9" containerID="40f2f72bc74aa8ccadb57321213d7742a3b0a4820875bd529594ebec0e2c3224" exitCode=0 Sep 08 14:16:49.457411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.457345 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" event={"ID":"40496547e3481087fc874b11f52284a9","Type":"ContainerDied","Data":"40f2f72bc74aa8ccadb57321213d7742a3b0a4820875bd529594ebec0e2c3224"} Sep 08 14:16:49.458949 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.458927 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xrx9d" event={"ID":"f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c","Type":"ContainerStarted","Data":"ef667cdf18a5d0e2dea063188bb1cfb3eaf044b83fd8d0fbed7a809dd0b799cc"} Sep 08 14:16:49.465762 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.465683 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vnl5t" podStartSLOduration=2.563614173 podStartE2EDuration="21.465671723s" totalImagesPullingTime="18.90205755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:16:49.465442292 +0000 UTC m=+21.749759408" watchObservedRunningTime="2026-09-08 14:16:49.465671723 +0000 UTC m=+21.749988837" Sep 08 14:16:49.495354 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.495317 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hlpbr" podStartSLOduration=2.510949938 podStartE2EDuration="21.495305263s" totalImagesPullingTime="18.984355325s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:16:49.494895216 +0000 UTC m=+21.779212331" watchObservedRunningTime="2026-09-08 14:16:49.495305263 +0000 UTC m=+21.779622378" Sep 08 14:16:49.517984 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.517916 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wrvcw" podStartSLOduration=2.007124108 podStartE2EDuration="21.517903851s" totalImagesPullingTime="19.510779743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:16:49.511556985 +0000 UTC m=+21.795874100" watchObservedRunningTime="2026-09-08 14:16:49.517903851 +0000 UTC m=+21.802220965" Sep 08 14:16:49.526067 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.526037 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tpxxk" podStartSLOduration=2.63021991 podStartE2EDuration="21.526030026s" totalImagesPullingTime="18.895810116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:16:49.525864769 +0000 UTC m=+21.810181883" watchObservedRunningTime="2026-09-08 14:16:49.526030026 +0000 UTC m=+21.810347140" Sep 08 14:16:49.559931 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.559897 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-155.ec2.internal" podStartSLOduration=20.559885453 podStartE2EDuration="20.559885453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:16:49.559832746 +0000 UTC m=+21.844149860" watchObservedRunningTime="2026-09-08 14:16:49.559885453 +0000 UTC m=+21.844202597" Sep 08 14:16:49.665007 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.664976 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tpxxk" Sep 08 14:16:49.665552 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.665533 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tpxxk" Sep 08 14:16:49.684932 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:49.684890 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xrx9d" podStartSLOduration=2.425295049 podStartE2EDuration="21.684875424s" totalImagesPullingTime="19.259580375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-08 14:16:49.574180393 +0000 UTC m=+21.858497507" watchObservedRunningTime="2026-09-08 14:16:49.684875424 +0000 UTC m=+21.969192520" Sep 08 14:16:50.235862 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:50.235659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:50.236000 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:50.235659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:50.236000 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:50.235958 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:50.236084 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:50.235996 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:50.463774 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:50.463667 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pxmln" event={"ID":"cf954761-020d-4a65-8997-5b4e59b79748","Type":"ContainerStarted","Data":"8e98d8edb2e2af3f11a5bdf642362ff6c46ce33a341ac0b8ff63f7ec0d5723f6"} Sep 08 14:16:50.465870 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:50.465836 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" event={"ID":"40496547e3481087fc874b11f52284a9","Type":"ContainerStarted","Data":"e733f37a7842492f22ad2699171b7ab53e74c3be3ee48c6db2aada247f0574a0"} Sep 08 14:16:50.466583 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:50.466561 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tpxxk" Sep 08 14:16:50.467155 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:50.467129 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tpxxk" Sep 08 14:16:50.478545 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:50.478507 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pxmln" podStartSLOduration=3.505087904 podStartE2EDuration="22.478490893s" totalImagesPullingTime="18.973402989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:16:50.478259955 +0000 UTC m=+22.762577070" watchObservedRunningTime="2026-09-08 14:16:50.478490893 +0000 UTC m=+22.762808009" Sep 08 14:16:50.493771 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:50.493713 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-155.ec2.internal" podStartSLOduration=21.493698421 podStartE2EDuration="21.493698421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:16:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:16:50.493641537 +0000 UTC m=+22.777958652" watchObservedRunningTime="2026-09-08 14:16:50.493698421 +0000 UTC m=+22.778015536" Sep 08 14:16:51.235380 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:51.235353 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:51.235538 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:51.235454 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:51.285023 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:51.285002 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 08 14:16:51.470947 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:51.470920 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" event={"ID":"df4f4a97-b500-41f7-bc90-dca2520927f6","Type":"ContainerStarted","Data":"11cd9381ea0a5756c07c17960d21b6834449616c5747ecadbe470a54087ecad7"} Sep 08 14:16:52.175902 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:52.175801 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T14:16:51.285018221Z","UUID":"ee8dd8bd-8c3d-44f7-b5b0-ac4ee81f8d55","Handler":null,"Name":"","Endpoint":""} Sep 08 14:16:52.177322 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:52.177306 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 08 14:16:52.177397 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:52.177336 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 08 14:16:52.235992 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:52.235961 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:52.236140 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:52.236083 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:52.236186 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:52.236137 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:52.236237 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:52.236219 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:52.477283 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:52.477209 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:16:52.477670 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:52.477641 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" event={"ID":"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397","Type":"ContainerStarted","Data":"7b07dec747b8eb714da57e261835c6c0eb46d950163ed607963c45779c861f97"} Sep 08 14:16:53.235178 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:53.235143 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:53.235356 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:53.235264 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:54.238482 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.238287 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:54.238884 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.238291 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:54.238884 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:54.238514 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:54.238884 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:54.238610 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:54.484113 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.484078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" event={"ID":"df4f4a97-b500-41f7-bc90-dca2520927f6","Type":"ContainerStarted","Data":"fa6adb906407c5115c9601d21462db412603b4e1b033b5854eb1e5682f2bd1a4"} Sep 08 14:16:54.487497 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.487476 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:16:54.487827 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.487804 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" event={"ID":"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397","Type":"ContainerStarted","Data":"a616ca66043a3e463a331dc8b7ee79fcdb04a1305e53b44fd0d81e479cf0ec63"} Sep 08 14:16:54.488020 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.488007 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:54.488095 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.488052 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:54.488277 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.488235 2578 scope.go:117] "RemoveContainer" containerID="132ddbd85d9e99ffd3f2e0678de8c41a2a11ed651c89eb1173c43db49417c670" Sep 08 14:16:54.489918 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.489895 2578 generic.go:356] "Generic (PLEG): container finished" podID="9d5bf364-23e4-42d1-b576-92465114d3c8" containerID="98d3e47bedeb02e2945dab28cdfd8766fa6ee4979cbae3867f84dec1dbfc2e00" exitCode=0 Sep 08 14:16:54.490032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.489930 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kt9z" event={"ID":"9d5bf364-23e4-42d1-b576-92465114d3c8","Type":"ContainerDied","Data":"98d3e47bedeb02e2945dab28cdfd8766fa6ee4979cbae3867f84dec1dbfc2e00"} Sep 08 14:16:54.504880 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.504858 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:54.506718 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.506702 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:54.513611 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.513568 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgfg2" podStartSLOduration=3.12632873 podStartE2EDuration="26.513555415s" totalImagesPullingTime="23.387226685s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:16:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:16:54.510670625 +0000 UTC m=+26.794987740" watchObservedRunningTime="2026-09-08 14:16:54.513555415 +0000 UTC m=+26.797872524" Sep 08 14:16:54.897453 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:54.897427 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:16:55.236029 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:55.235942 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:55.236188 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:55.236082 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:55.495473 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:55.495415 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:16:55.496047 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:55.495833 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" event={"ID":"1a3ac79a-e3b0-4b49-aa08-d77e1d12e397","Type":"ContainerStarted","Data":"cca73215b157017dd66d3591afaadd367595e57855711de112b34c17ccf5c02a"} Sep 08 14:16:55.523887 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:55.523833 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" podStartSLOduration=8.473443917 podStartE2EDuration="27.523812708s" totalImagesPullingTime="19.050368791s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:16:55.522999978 +0000 UTC m=+27.807317094" watchObservedRunningTime="2026-09-08 14:16:55.523812708 +0000 UTC m=+27.808129825" Sep 08 14:16:55.857226 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:55.857161 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hsh78"] Sep 08 14:16:55.857376 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:55.857271 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:55.857430 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:55.857386 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:55.860048 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:55.860023 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4njj8"] Sep 08 14:16:55.860175 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:55.860104 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:55.860175 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:55.860162 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:55.862064 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:55.862041 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-998rf"] Sep 08 14:16:55.862199 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:55.862139 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:55.862291 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:55.862249 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:57.235637 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:57.235455 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:57.236182 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:57.235520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:57.236182 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:57.235770 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:57.236182 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:57.235797 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:57.503357 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:57.503272 2578 generic.go:356] "Generic (PLEG): container finished" podID="9d5bf364-23e4-42d1-b576-92465114d3c8" containerID="079c0c9f5dc30321bf46b2385dfc206f0a2fd04f760be2fe2fa6105b5c313a99" exitCode=0 Sep 08 14:16:57.503357 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:57.503325 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kt9z" event={"ID":"9d5bf364-23e4-42d1-b576-92465114d3c8","Type":"ContainerDied","Data":"079c0c9f5dc30321bf46b2385dfc206f0a2fd04f760be2fe2fa6105b5c313a99"} Sep 08 14:16:58.235979 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:58.235946 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:16:58.236286 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:58.236028 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:16:59.235281 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:59.235259 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:16:59.235392 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:59.235262 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:16:59.235445 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:59.235384 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:16:59.235445 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:16:59.235427 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:16:59.510525 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:59.510444 2578 generic.go:356] "Generic (PLEG): container finished" podID="9d5bf364-23e4-42d1-b576-92465114d3c8" containerID="6e45a9ffbc050ef37bcbc2ffe3c82ef98d9dd6d7e1553966de1885fba6ff66ff" exitCode=0 Sep 08 14:16:59.510525 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:16:59.510512 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kt9z" event={"ID":"9d5bf364-23e4-42d1-b576-92465114d3c8","Type":"ContainerDied","Data":"6e45a9ffbc050ef37bcbc2ffe3c82ef98d9dd6d7e1553966de1885fba6ff66ff"} Sep 08 14:17:00.235458 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:00.235424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:17:00.236081 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:00.235565 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4njj8" podUID="43ad5ec2-69c5-4c3a-a997-adb20472335a" Sep 08 14:17:00.853049 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:00.853005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs\") pod \"network-metrics-daemon-998rf\" (UID: \"5267f2dc-393e-40f2-8b78-25179961c1a3\") " pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:17:00.853519 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:00.853069 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:17:00.853519 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:00.853163 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 08 14:17:00.853519 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:00.853165 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:17:00.853519 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:00.853218 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret podName:b4afcd3a-ef6b-4488-89b7-c48529821f6c nodeName:}" failed. No retries permitted until 2026-09-08 14:17:32.853201025 +0000 UTC m=+65.137518119 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret") pod "global-pull-secret-syncer-hsh78" (UID: "b4afcd3a-ef6b-4488-89b7-c48529821f6c") : object "kube-system"/"original-pull-secret" not registered Sep 08 14:17:00.853519 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:00.853234 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs podName:5267f2dc-393e-40f2-8b78-25179961c1a3 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:32.853226731 +0000 UTC m=+65.137543824 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs") pod "network-metrics-daemon-998rf" (UID: "5267f2dc-393e-40f2-8b78-25179961c1a3") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 08 14:17:00.953877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:00.953835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8d5w\" (UniqueName: \"kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w\") pod \"network-check-target-4njj8\" (UID: \"43ad5ec2-69c5-4c3a-a997-adb20472335a\") " pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:17:00.954055 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:00.954012 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 08 14:17:00.954055 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:00.954043 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 08 14:17:00.954140 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:00.954057 2578 projected.go:196] Error preparing data for projected volume kube-api-access-f8d5w for pod openshift-network-diagnostics/network-check-target-4njj8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:17:00.954140 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:00.954124 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w podName:43ad5ec2-69c5-4c3a-a997-adb20472335a nodeName:}" failed. No retries permitted until 2026-09-08 14:17:32.954104071 +0000 UTC m=+65.238421165 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-f8d5w" (UniqueName: "kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w") pod "network-check-target-4njj8" (UID: "43ad5ec2-69c5-4c3a-a997-adb20472335a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 08 14:17:01.235488 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:01.235388 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:17:01.235488 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:01.235430 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:17:01.235718 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:01.235530 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-998rf" podUID="5267f2dc-393e-40f2-8b78-25179961c1a3" Sep 08 14:17:01.235718 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:01.235666 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsh78" podUID="b4afcd3a-ef6b-4488-89b7-c48529821f6c" Sep 08 14:17:01.984224 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:01.984150 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-155.ec2.internal" event="NodeReady" Sep 08 14:17:01.984581 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:01.984347 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 08 14:17:02.040247 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.040202 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4wnjb"] Sep 08 14:17:02.042387 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.042363 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pxs2b"] Sep 08 14:17:02.042651 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.042629 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:02.044320 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.044301 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pxs2b" Sep 08 14:17:02.051790 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.051773 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 08 14:17:02.057206 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.057049 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 08 14:17:02.057206 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.057079 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 08 14:17:02.057206 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.057142 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 08 14:17:02.057206 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.057144 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 08 14:17:02.057425 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.057405 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rlxm6\"" Sep 08 14:17:02.057631 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.057608 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-j8jbf\"" Sep 08 14:17:02.062329 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.062310 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pxs2b"] Sep 08 14:17:02.071899 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.069636 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4wnjb"] Sep 08 14:17:02.161518 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.161483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96c69\" (UniqueName: \"kubernetes.io/projected/38343840-71de-4d89-b76f-d628180edba4-kube-api-access-96c69\") pod \"ingress-canary-pxs2b\" (UID: \"38343840-71de-4d89-b76f-d628180edba4\") " pod="openshift-ingress-canary/ingress-canary-pxs2b" Sep 08 14:17:02.161690 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.161556 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpr4x\" (UniqueName: \"kubernetes.io/projected/fa59392d-7939-41be-bcb6-0a382dfa8583-kube-api-access-xpr4x\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:02.161690 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.161620 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa59392d-7939-41be-bcb6-0a382dfa8583-tmp-dir\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:02.161690 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.161663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:02.161850 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.161691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38343840-71de-4d89-b76f-d628180edba4-cert\") pod \"ingress-canary-pxs2b\" (UID: \"38343840-71de-4d89-b76f-d628180edba4\") " pod="openshift-ingress-canary/ingress-canary-pxs2b" Sep 08 14:17:02.161888 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.161858 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa59392d-7939-41be-bcb6-0a382dfa8583-config-volume\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:02.235632 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.235559 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:17:02.238629 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.238411 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vvt7d\"" Sep 08 14:17:02.238629 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.238411 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 14:17:02.238629 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.238539 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 14:17:02.262790 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.262761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa59392d-7939-41be-bcb6-0a382dfa8583-config-volume\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:02.262935 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.262825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96c69\" (UniqueName: \"kubernetes.io/projected/38343840-71de-4d89-b76f-d628180edba4-kube-api-access-96c69\") pod \"ingress-canary-pxs2b\" (UID: \"38343840-71de-4d89-b76f-d628180edba4\") " pod="openshift-ingress-canary/ingress-canary-pxs2b" Sep 08 14:17:02.262935 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.262875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpr4x\" (UniqueName: \"kubernetes.io/projected/fa59392d-7939-41be-bcb6-0a382dfa8583-kube-api-access-xpr4x\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:02.262935 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.262904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa59392d-7939-41be-bcb6-0a382dfa8583-tmp-dir\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:02.262935 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.262928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:02.263144 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.262955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38343840-71de-4d89-b76f-d628180edba4-cert\") pod \"ingress-canary-pxs2b\" (UID: \"38343840-71de-4d89-b76f-d628180edba4\") " pod="openshift-ingress-canary/ingress-canary-pxs2b" Sep 08 14:17:02.263241 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:02.263223 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 14:17:02.263297 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.263281 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa59392d-7939-41be-bcb6-0a382dfa8583-tmp-dir\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:02.263346 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:02.263289 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls podName:fa59392d-7939-41be-bcb6-0a382dfa8583 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:02.763269422 +0000 UTC m=+35.047586529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls") pod "dns-default-4wnjb" (UID: "fa59392d-7939-41be-bcb6-0a382dfa8583") : secret "dns-default-metrics-tls" not found Sep 08 14:17:02.263551 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.263533 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fa59392d-7939-41be-bcb6-0a382dfa8583-config-volume\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:02.268098 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.267980 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38343840-71de-4d89-b76f-d628180edba4-cert\") pod \"ingress-canary-pxs2b\" (UID: \"38343840-71de-4d89-b76f-d628180edba4\") " pod="openshift-ingress-canary/ingress-canary-pxs2b" Sep 08 14:17:02.275136 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.275113 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpr4x\" (UniqueName: \"kubernetes.io/projected/fa59392d-7939-41be-bcb6-0a382dfa8583-kube-api-access-xpr4x\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:02.275324 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.275305 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96c69\" (UniqueName: \"kubernetes.io/projected/38343840-71de-4d89-b76f-d628180edba4-kube-api-access-96c69\") pod \"ingress-canary-pxs2b\" (UID: \"38343840-71de-4d89-b76f-d628180edba4\") " pod="openshift-ingress-canary/ingress-canary-pxs2b" Sep 08 14:17:02.361511 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.361476 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pxs2b" Sep 08 14:17:02.652025 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.651959 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pxs2b"] Sep 08 14:17:02.757570 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.757539 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-tbsgb"] Sep 08 14:17:02.760533 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.760510 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:02.761906 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.761883 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw"] Sep 08 14:17:02.763480 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.763461 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5bs7t\"" Sep 08 14:17:02.763578 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.763558 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57556ff886-kmtht"] Sep 08 14:17:02.763578 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.763468 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 08 14:17:02.763789 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.763774 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:02.764285 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.764256 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 08 14:17:02.764414 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.764280 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 08 14:17:02.764609 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.764470 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 08 14:17:02.765368 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.765349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.766718 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.766700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:02.766908 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:02.766878 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 14:17:02.767003 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:02.766941 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls podName:fa59392d-7939-41be-bcb6-0a382dfa8583 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:03.766923964 +0000 UTC m=+36.051241078 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls") pod "dns-default-4wnjb" (UID: "fa59392d-7939-41be-bcb6-0a382dfa8583") : secret "dns-default-metrics-tls" not found Sep 08 14:17:02.768526 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.768416 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 08 14:17:02.768680 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.768589 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 08 14:17:02.768949 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.768930 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 08 14:17:02.768949 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.768942 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 08 14:17:02.769087 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.768966 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m9ct5\"" Sep 08 14:17:02.769319 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.769296 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 08 14:17:02.769898 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.769626 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kj4kg\"" Sep 08 14:17:02.770087 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.770072 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 08 14:17:02.770428 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.770410 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 08 14:17:02.773876 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.773819 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 08 14:17:02.773971 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.773948 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-tbsgb"] Sep 08 14:17:02.774523 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.774503 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw"] Sep 08 14:17:02.777244 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.777229 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 08 14:17:02.778931 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.778911 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57556ff886-kmtht"] Sep 08 14:17:02.866688 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.866656 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2cfl"] Sep 08 14:17:02.867406 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867377 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.867539 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867415 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-bound-sa-token\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.867539 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867448 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qmt\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-kube-api-access-t2qmt\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.867539 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867510 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7facf738-68e8-45a4-b6a1-6758b93a47ed-config\") pod \"console-operator-85b454874b-tbsgb\" (UID: \"7facf738-68e8-45a4-b6a1-6758b93a47ed\") " pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:02.867720 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867543 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7facf738-68e8-45a4-b6a1-6758b93a47ed-serving-cert\") pod \"console-operator-85b454874b-tbsgb\" (UID: \"7facf738-68e8-45a4-b6a1-6758b93a47ed\") " pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:02.867720 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c5006d65-0045-4f1a-9946-f46325714d52-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:02.867720 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867591 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhx7r\" (UniqueName: \"kubernetes.io/projected/c5006d65-0045-4f1a-9946-f46325714d52-kube-api-access-bhx7r\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:02.867720 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b31922ec-e476-4ca8-9d58-e64859608a90-registry-certificates\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.867720 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b31922ec-e476-4ca8-9d58-e64859608a90-ca-trust-extracted\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.867720 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqn7b\" (UniqueName: \"kubernetes.io/projected/7facf738-68e8-45a4-b6a1-6758b93a47ed-kube-api-access-jqn7b\") pod \"console-operator-85b454874b-tbsgb\" (UID: \"7facf738-68e8-45a4-b6a1-6758b93a47ed\") " pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:02.867720 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867710 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b31922ec-e476-4ca8-9d58-e64859608a90-installation-pull-secrets\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.868004 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867759 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b31922ec-e476-4ca8-9d58-e64859608a90-image-registry-private-configuration\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.868004 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b31922ec-e476-4ca8-9d58-e64859608a90-trusted-ca\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.868004 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867812 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7facf738-68e8-45a4-b6a1-6758b93a47ed-trusted-ca\") pod \"console-operator-85b454874b-tbsgb\" (UID: \"7facf738-68e8-45a4-b6a1-6758b93a47ed\") " pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:02.868004 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.867860 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:02.868703 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.868683 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-fpwss"] Sep 08 14:17:02.868844 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.868798 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2cfl" Sep 08 14:17:02.870582 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.870565 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:02.871154 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.871133 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 08 14:17:02.871265 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.871167 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 08 14:17:02.871265 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.871253 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2448s\"" Sep 08 14:17:02.872578 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.872558 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 08 14:17:02.872704 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.872684 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 08 14:17:02.872874 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.872858 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 08 14:17:02.872944 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.872865 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 08 14:17:02.873151 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.873135 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lbhnd\"" Sep 08 14:17:02.878740 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.878693 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 08 14:17:02.883042 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.883016 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-fpwss"] Sep 08 14:17:02.884037 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.884014 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2cfl"] Sep 08 14:17:02.965149 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.965068 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-2x4fm"] Sep 08 14:17:02.967061 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.967044 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:02.968247 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968199 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-tmp\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:02.968247 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.968247 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968246 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-bound-sa-token\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.968489 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtvn8\" (UniqueName: \"kubernetes.io/projected/f3b66b59-514f-4b58-abed-5b1d698c556e-kube-api-access-rtvn8\") pod \"volume-data-source-validator-695c895bd6-r2cfl\" (UID: \"f3b66b59-514f-4b58-abed-5b1d698c556e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2cfl" Sep 08 14:17:02.968489 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qmt\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-kube-api-access-t2qmt\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.968489 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:02.968372 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 14:17:02.968489 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7facf738-68e8-45a4-b6a1-6758b93a47ed-config\") pod \"console-operator-85b454874b-tbsgb\" (UID: \"7facf738-68e8-45a4-b6a1-6758b93a47ed\") " pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:02.968489 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:02.968392 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57556ff886-kmtht: secret "image-registry-tls" not found Sep 08 14:17:02.968489 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7facf738-68e8-45a4-b6a1-6758b93a47ed-serving-cert\") pod \"console-operator-85b454874b-tbsgb\" (UID: \"7facf738-68e8-45a4-b6a1-6758b93a47ed\") " pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:02.968489 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c5006d65-0045-4f1a-9946-f46325714d52-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:02.968489 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:02.968449 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls podName:b31922ec-e476-4ca8-9d58-e64859608a90 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:03.468430683 +0000 UTC m=+35.752747781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls") pod "image-registry-57556ff886-kmtht" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90") : secret "image-registry-tls" not found Sep 08 14:17:02.968489 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhx7r\" (UniqueName: \"kubernetes.io/projected/c5006d65-0045-4f1a-9946-f46325714d52-kube-api-access-bhx7r\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:02.969032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b31922ec-e476-4ca8-9d58-e64859608a90-registry-certificates\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.969032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b31922ec-e476-4ca8-9d58-e64859608a90-ca-trust-extracted\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.969032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:02.969032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968640 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqn7b\" (UniqueName: \"kubernetes.io/projected/7facf738-68e8-45a4-b6a1-6758b93a47ed-kube-api-access-jqn7b\") pod \"console-operator-85b454874b-tbsgb\" (UID: \"7facf738-68e8-45a4-b6a1-6758b93a47ed\") " pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:02.969032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b31922ec-e476-4ca8-9d58-e64859608a90-installation-pull-secrets\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.969032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b31922ec-e476-4ca8-9d58-e64859608a90-image-registry-private-configuration\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.969032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968721 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b31922ec-e476-4ca8-9d58-e64859608a90-trusted-ca\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.969032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968770 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-serving-cert\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:02.969032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7facf738-68e8-45a4-b6a1-6758b93a47ed-trusted-ca\") pod \"console-operator-85b454874b-tbsgb\" (UID: \"7facf738-68e8-45a4-b6a1-6758b93a47ed\") " pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:02.969032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968842 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-snapshots\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:02.969032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968868 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:02.969032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzrwc\" (UniqueName: \"kubernetes.io/projected/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-kube-api-access-dzrwc\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:02.969032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.968931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:02.969861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.969042 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c5006d65-0045-4f1a-9946-f46325714d52-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:02.969861 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:02.969052 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 14:17:02.969861 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:02.969098 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls podName:c5006d65-0045-4f1a-9946-f46325714d52 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:03.469083784 +0000 UTC m=+35.753400890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-d9tcw" (UID: "c5006d65-0045-4f1a-9946-f46325714d52") : secret "cluster-monitoring-operator-tls" not found Sep 08 14:17:02.969861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.969435 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7facf738-68e8-45a4-b6a1-6758b93a47ed-config\") pod \"console-operator-85b454874b-tbsgb\" (UID: \"7facf738-68e8-45a4-b6a1-6758b93a47ed\") " pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:02.969861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.969443 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kkhk5\"" Sep 08 14:17:02.969861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.969634 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 08 14:17:02.969861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.969762 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b31922ec-e476-4ca8-9d58-e64859608a90-ca-trust-extracted\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.969861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.969857 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 08 14:17:02.970252 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.969931 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b31922ec-e476-4ca8-9d58-e64859608a90-registry-certificates\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.970252 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.970010 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b31922ec-e476-4ca8-9d58-e64859608a90-trusted-ca\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.971414 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.971375 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7facf738-68e8-45a4-b6a1-6758b93a47ed-trusted-ca\") pod \"console-operator-85b454874b-tbsgb\" (UID: \"7facf738-68e8-45a4-b6a1-6758b93a47ed\") " pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:02.972360 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.972340 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b31922ec-e476-4ca8-9d58-e64859608a90-installation-pull-secrets\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.972549 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.972521 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7facf738-68e8-45a4-b6a1-6758b93a47ed-serving-cert\") pod \"console-operator-85b454874b-tbsgb\" (UID: \"7facf738-68e8-45a4-b6a1-6758b93a47ed\") " pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:02.973563 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.973541 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b31922ec-e476-4ca8-9d58-e64859608a90-image-registry-private-configuration\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.977099 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.977045 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-2x4fm"] Sep 08 14:17:02.981670 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.981575 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhx7r\" (UniqueName: \"kubernetes.io/projected/c5006d65-0045-4f1a-9946-f46325714d52-kube-api-access-bhx7r\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:02.982109 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.982087 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqn7b\" (UniqueName: \"kubernetes.io/projected/7facf738-68e8-45a4-b6a1-6758b93a47ed-kube-api-access-jqn7b\") pod \"console-operator-85b454874b-tbsgb\" (UID: \"7facf738-68e8-45a4-b6a1-6758b93a47ed\") " pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:02.982184 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.982093 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qmt\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-kube-api-access-t2qmt\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:02.982184 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:02.982119 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-bound-sa-token\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:03.069593 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.069560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-tmp\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:03.070419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.069623 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-serving-cert\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:03.070419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.069652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2x4fm\" (UID: \"9c8ce093-b801-4390-b647-cd3ea9430ab4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:03.070419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.069673 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:03.070419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.069691 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzrwc\" (UniqueName: \"kubernetes.io/projected/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-kube-api-access-dzrwc\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:03.070419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.069916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c8ce093-b801-4390-b647-cd3ea9430ab4-nginx-conf\") pod \"networking-console-plugin-79977f9844-2x4fm\" (UID: \"9c8ce093-b801-4390-b647-cd3ea9430ab4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:03.070419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.069955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtvn8\" (UniqueName: \"kubernetes.io/projected/f3b66b59-514f-4b58-abed-5b1d698c556e-kube-api-access-rtvn8\") pod \"volume-data-source-validator-695c895bd6-r2cfl\" (UID: \"f3b66b59-514f-4b58-abed-5b1d698c556e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2cfl" Sep 08 14:17:03.070419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.069999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:03.070419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.070027 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-tmp\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:03.070419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.070036 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-snapshots\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:03.070711 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.070670 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:03.070711 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.070685 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-snapshots\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:03.071234 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.071212 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:03.072970 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.072951 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-serving-cert\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:03.073856 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.073836 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:03.078623 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.078596 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzrwc\" (UniqueName: \"kubernetes.io/projected/bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55-kube-api-access-dzrwc\") pod \"insights-operator-76bcd6ccf5-fpwss\" (UID: \"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55\") " pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:03.079827 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.079809 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtvn8\" (UniqueName: \"kubernetes.io/projected/f3b66b59-514f-4b58-abed-5b1d698c556e-kube-api-access-rtvn8\") pod \"volume-data-source-validator-695c895bd6-r2cfl\" (UID: \"f3b66b59-514f-4b58-abed-5b1d698c556e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2cfl" Sep 08 14:17:03.170816 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.170786 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2x4fm\" (UID: \"9c8ce093-b801-4390-b647-cd3ea9430ab4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:03.170994 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.170850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c8ce093-b801-4390-b647-cd3ea9430ab4-nginx-conf\") pod \"networking-console-plugin-79977f9844-2x4fm\" (UID: \"9c8ce093-b801-4390-b647-cd3ea9430ab4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:03.170994 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:03.170928 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 14:17:03.171116 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:03.171007 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert podName:9c8ce093-b801-4390-b647-cd3ea9430ab4 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:03.670987801 +0000 UTC m=+35.955304895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-2x4fm" (UID: "9c8ce093-b801-4390-b647-cd3ea9430ab4") : secret "networking-console-plugin-cert" not found Sep 08 14:17:03.171528 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.171505 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c8ce093-b801-4390-b647-cd3ea9430ab4-nginx-conf\") pod \"networking-console-plugin-79977f9844-2x4fm\" (UID: \"9c8ce093-b801-4390-b647-cd3ea9430ab4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:03.182252 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.182227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2cfl" Sep 08 14:17:03.188186 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.188162 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" Sep 08 14:17:03.235337 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.235280 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:17:03.235462 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.235446 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:17:03.238134 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.237850 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 14:17:03.238134 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.237866 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zbnn7\"" Sep 08 14:17:03.238134 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.237916 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 14:17:03.473878 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.473844 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:03.474041 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.473962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:03.474041 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:03.474021 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 14:17:03.474155 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:03.474083 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 14:17:03.474155 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:03.474098 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57556ff886-kmtht: secret "image-registry-tls" not found Sep 08 14:17:03.474155 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:03.474110 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls podName:c5006d65-0045-4f1a-9946-f46325714d52 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:04.474088638 +0000 UTC m=+36.758405734 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-d9tcw" (UID: "c5006d65-0045-4f1a-9946-f46325714d52") : secret "cluster-monitoring-operator-tls" not found Sep 08 14:17:03.474753 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:03.474474 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls podName:b31922ec-e476-4ca8-9d58-e64859608a90 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:04.474131623 +0000 UTC m=+36.758448736 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls") pod "image-registry-57556ff886-kmtht" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90") : secret "image-registry-tls" not found Sep 08 14:17:03.675508 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.675425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2x4fm\" (UID: \"9c8ce093-b801-4390-b647-cd3ea9430ab4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:03.675650 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:03.675595 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 14:17:03.675698 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:03.675689 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert podName:9c8ce093-b801-4390-b647-cd3ea9430ab4 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:04.67566535 +0000 UTC m=+36.959982506 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-2x4fm" (UID: "9c8ce093-b801-4390-b647-cd3ea9430ab4") : secret "networking-console-plugin-cert" not found Sep 08 14:17:03.776319 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:03.776279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:03.776482 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:03.776432 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 14:17:03.776529 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:03.776505 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls podName:fa59392d-7939-41be-bcb6-0a382dfa8583 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:05.776484177 +0000 UTC m=+38.060801280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls") pod "dns-default-4wnjb" (UID: "fa59392d-7939-41be-bcb6-0a382dfa8583") : secret "dns-default-metrics-tls" not found Sep 08 14:17:04.144615 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:04.144533 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38343840_71de_4d89_b76f_d628180edba4.slice/crio-7b2e03a704f45a897ef5b0799edc3811c261cad90e7009fa962b73ee6b2380e6 WatchSource:0}: Error finding container 7b2e03a704f45a897ef5b0799edc3811c261cad90e7009fa962b73ee6b2380e6: Status 404 returned error can't find the container with id 7b2e03a704f45a897ef5b0799edc3811c261cad90e7009fa962b73ee6b2380e6 Sep 08 14:17:04.426968 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:04.426750 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-fpwss"] Sep 08 14:17:04.431300 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:04.431274 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2cfl"] Sep 08 14:17:04.432099 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:04.432079 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-tbsgb"] Sep 08 14:17:04.442367 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:04.442337 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7facf738_68e8_45a4_b6a1_6758b93a47ed.slice/crio-946e8c83ac5f922c7b42799b297ce07fb73902f6f1f32e1c10baf789baf2ca4b WatchSource:0}: Error finding container 946e8c83ac5f922c7b42799b297ce07fb73902f6f1f32e1c10baf789baf2ca4b: Status 404 returned error can't find the container with id 946e8c83ac5f922c7b42799b297ce07fb73902f6f1f32e1c10baf789baf2ca4b Sep 08 14:17:04.442942 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:04.442669 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd3f6ba_5d2f_4300_8fdd_f0dd38904e55.slice/crio-dd9b422467cc68d815aef1a43fe79c7dea39c59effd511ea94de871c1975e8b5 WatchSource:0}: Error finding container dd9b422467cc68d815aef1a43fe79c7dea39c59effd511ea94de871c1975e8b5: Status 404 returned error can't find the container with id dd9b422467cc68d815aef1a43fe79c7dea39c59effd511ea94de871c1975e8b5 Sep 08 14:17:04.442998 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:04.442952 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b66b59_514f_4b58_abed_5b1d698c556e.slice/crio-cb7ca7428f315d95981cffb0e10b66a328f98af7eb56f7b9f7b0bac23c90ba39 WatchSource:0}: Error finding container cb7ca7428f315d95981cffb0e10b66a328f98af7eb56f7b9f7b0bac23c90ba39: Status 404 returned error can't find the container with id cb7ca7428f315d95981cffb0e10b66a328f98af7eb56f7b9f7b0bac23c90ba39 Sep 08 14:17:04.482591 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:04.482567 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:04.482683 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:04.482656 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:04.482773 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:04.482753 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 14:17:04.482773 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:04.482767 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 14:17:04.482875 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:04.482781 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57556ff886-kmtht: secret "image-registry-tls" not found Sep 08 14:17:04.482875 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:04.482830 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls podName:c5006d65-0045-4f1a-9946-f46325714d52 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:06.48280953 +0000 UTC m=+38.767126626 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-d9tcw" (UID: "c5006d65-0045-4f1a-9946-f46325714d52") : secret "cluster-monitoring-operator-tls" not found Sep 08 14:17:04.482875 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:04.482851 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls podName:b31922ec-e476-4ca8-9d58-e64859608a90 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:06.48284131 +0000 UTC m=+38.767158404 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls") pod "image-registry-57556ff886-kmtht" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90") : secret "image-registry-tls" not found Sep 08 14:17:04.525885 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:04.525849 2578 generic.go:356] "Generic (PLEG): container finished" podID="9d5bf364-23e4-42d1-b576-92465114d3c8" containerID="0b9270a81ecb8180bb72bf45c12ad045d582b244006ed3f3d59ea7b057eda2a1" exitCode=0 Sep 08 14:17:04.526010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:04.525929 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kt9z" event={"ID":"9d5bf364-23e4-42d1-b576-92465114d3c8","Type":"ContainerDied","Data":"0b9270a81ecb8180bb72bf45c12ad045d582b244006ed3f3d59ea7b057eda2a1"} Sep 08 14:17:04.527125 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:04.527100 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2cfl" event={"ID":"f3b66b59-514f-4b58-abed-5b1d698c556e","Type":"ContainerStarted","Data":"cb7ca7428f315d95981cffb0e10b66a328f98af7eb56f7b9f7b0bac23c90ba39"} Sep 08 14:17:04.528195 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:04.528175 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" event={"ID":"7facf738-68e8-45a4-b6a1-6758b93a47ed","Type":"ContainerStarted","Data":"946e8c83ac5f922c7b42799b297ce07fb73902f6f1f32e1c10baf789baf2ca4b"} Sep 08 14:17:04.529438 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:04.529414 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" event={"ID":"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55","Type":"ContainerStarted","Data":"dd9b422467cc68d815aef1a43fe79c7dea39c59effd511ea94de871c1975e8b5"} Sep 08 14:17:04.530583 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:04.530566 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pxs2b" event={"ID":"38343840-71de-4d89-b76f-d628180edba4","Type":"ContainerStarted","Data":"7b2e03a704f45a897ef5b0799edc3811c261cad90e7009fa962b73ee6b2380e6"} Sep 08 14:17:04.684320 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:04.684077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2x4fm\" (UID: \"9c8ce093-b801-4390-b647-cd3ea9430ab4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:04.684320 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:04.684229 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 14:17:04.684824 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:04.684809 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert podName:9c8ce093-b801-4390-b647-cd3ea9430ab4 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:06.684789539 +0000 UTC m=+38.969106635 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-2x4fm" (UID: "9c8ce093-b801-4390-b647-cd3ea9430ab4") : secret "networking-console-plugin-cert" not found Sep 08 14:17:05.541500 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:05.540754 2578 generic.go:356] "Generic (PLEG): container finished" podID="9d5bf364-23e4-42d1-b576-92465114d3c8" containerID="28361de66173c1c77c968ba05745fa43cb198089a6a08fb09887619e5d159da1" exitCode=0 Sep 08 14:17:05.541500 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:05.540825 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kt9z" event={"ID":"9d5bf364-23e4-42d1-b576-92465114d3c8","Type":"ContainerDied","Data":"28361de66173c1c77c968ba05745fa43cb198089a6a08fb09887619e5d159da1"} Sep 08 14:17:05.794546 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:05.794463 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:05.794703 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:05.794631 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 14:17:05.794703 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:05.794690 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls podName:fa59392d-7939-41be-bcb6-0a382dfa8583 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:09.79467308 +0000 UTC m=+42.078990196 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls") pod "dns-default-4wnjb" (UID: "fa59392d-7939-41be-bcb6-0a382dfa8583") : secret "dns-default-metrics-tls" not found Sep 08 14:17:06.500788 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:06.500739 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:06.500971 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:06.500880 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 14:17:06.500971 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:06.500950 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls podName:c5006d65-0045-4f1a-9946-f46325714d52 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:10.500928067 +0000 UTC m=+42.785245160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-d9tcw" (UID: "c5006d65-0045-4f1a-9946-f46325714d52") : secret "cluster-monitoring-operator-tls" not found Sep 08 14:17:06.500971 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:06.500883 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:06.500971 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:06.500968 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 14:17:06.501188 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:06.500983 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57556ff886-kmtht: secret "image-registry-tls" not found Sep 08 14:17:06.501188 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:06.501033 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls podName:b31922ec-e476-4ca8-9d58-e64859608a90 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:10.50101578 +0000 UTC m=+42.785332892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls") pod "image-registry-57556ff886-kmtht" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90") : secret "image-registry-tls" not found Sep 08 14:17:06.547694 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:06.547661 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6kt9z" event={"ID":"9d5bf364-23e4-42d1-b576-92465114d3c8","Type":"ContainerStarted","Data":"87eeb3e555ab6903a77c92d9d66970f39cee58f7ea900321a3220c12c5e056bb"} Sep 08 14:17:06.569507 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:06.569457 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6kt9z" podStartSLOduration=6.846549795 podStartE2EDuration="38.569442484s" totalImagesPullingTime="31.722892689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:16:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:06.568868474 +0000 UTC m=+38.853185588" watchObservedRunningTime="2026-09-08 14:17:06.569442484 +0000 UTC m=+38.853759625" Sep 08 14:17:06.702831 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:06.702789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2x4fm\" (UID: \"9c8ce093-b801-4390-b647-cd3ea9430ab4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:06.702974 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:06.702907 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 14:17:06.702974 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:06.702971 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert podName:9c8ce093-b801-4390-b647-cd3ea9430ab4 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:10.702953422 +0000 UTC m=+42.987270538 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-2x4fm" (UID: "9c8ce093-b801-4390-b647-cd3ea9430ab4") : secret "networking-console-plugin-cert" not found Sep 08 14:17:09.556092 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:09.555981 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pxs2b" event={"ID":"38343840-71de-4d89-b76f-d628180edba4","Type":"ContainerStarted","Data":"dfd3cbe82a2cf774d027914989b8d640fdec43624612c854352f2f1f36db0b8e"} Sep 08 14:17:09.557639 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:09.557615 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2cfl" event={"ID":"f3b66b59-514f-4b58-abed-5b1d698c556e","Type":"ContainerStarted","Data":"fa2538c39a89c0d16a727945d64ea39ccf3db1c5c5426913127434759f89f7ce"} Sep 08 14:17:09.559138 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:09.559121 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/0.log" Sep 08 14:17:09.559223 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:09.559155 2578 generic.go:356] "Generic (PLEG): container finished" podID="7facf738-68e8-45a4-b6a1-6758b93a47ed" containerID="dd3ee329d5a27992ca67c2121f65fbfca367acb2dc1b04d34a9825f2220162bb" exitCode=255 Sep 08 14:17:09.559223 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:09.559213 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" event={"ID":"7facf738-68e8-45a4-b6a1-6758b93a47ed","Type":"ContainerDied","Data":"dd3ee329d5a27992ca67c2121f65fbfca367acb2dc1b04d34a9825f2220162bb"} Sep 08 14:17:09.559393 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:09.559375 2578 scope.go:117] "RemoveContainer" containerID="dd3ee329d5a27992ca67c2121f65fbfca367acb2dc1b04d34a9825f2220162bb" Sep 08 14:17:09.560818 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:09.560798 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" event={"ID":"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55","Type":"ContainerStarted","Data":"d99de94d63fd7a1a390fd30ddf41a8be84deb7346abbb59579a0a0d016c3a105"} Sep 08 14:17:09.576605 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:09.576568 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pxs2b" podStartSLOduration=2.724070546 podStartE2EDuration="7.576555607s" totalImagesPullingTime="4.852485061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:09.575801961 +0000 UTC m=+41.860119089" watchObservedRunningTime="2026-09-08 14:17:09.576555607 +0000 UTC m=+41.860872727" Sep 08 14:17:09.595468 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:09.595415 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" podStartSLOduration=3.039395684 podStartE2EDuration="7.595396826s" totalImagesPullingTime="4.556001142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:09.594061872 +0000 UTC m=+41.878378988" watchObservedRunningTime="2026-09-08 14:17:09.595396826 +0000 UTC m=+41.879713943" Sep 08 14:17:09.629479 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:09.629428 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-r2cfl" podStartSLOduration=3.073693881 podStartE2EDuration="7.629409106s" totalImagesPullingTime="4.555715225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:09.628268435 +0000 UTC m=+41.912585549" watchObservedRunningTime="2026-09-08 14:17:09.629409106 +0000 UTC m=+41.913726222" Sep 08 14:17:09.830677 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:09.830609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:09.830820 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:09.830799 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 14:17:09.830886 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:09.830876 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls podName:fa59392d-7939-41be-bcb6-0a382dfa8583 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:17.830855862 +0000 UTC m=+50.115172958 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls") pod "dns-default-4wnjb" (UID: "fa59392d-7939-41be-bcb6-0a382dfa8583") : secret "dns-default-metrics-tls" not found Sep 08 14:17:10.379181 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.379149 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f8dx7"] Sep 08 14:17:10.402153 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.402119 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8dx7"] Sep 08 14:17:10.402273 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.402243 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.404414 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.404391 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2wblk\"" Sep 08 14:17:10.404414 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.404407 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 08 14:17:10.404584 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.404407 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 08 14:17:10.536140 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.536112 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/91526aea-a23b-4e12-a739-4a4b9c73433d-data-volume\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.536345 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.536169 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.536345 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.536245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:10.536345 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.536290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.536489 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.536369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:10.536489 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:10.536394 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 14:17:10.536489 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:10.536413 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57556ff886-kmtht: secret "image-registry-tls" not found Sep 08 14:17:10.536489 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.536422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/91526aea-a23b-4e12-a739-4a4b9c73433d-crio-socket\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.536489 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:10.536452 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 14:17:10.536489 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:10.536470 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls podName:b31922ec-e476-4ca8-9d58-e64859608a90 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:18.536451362 +0000 UTC m=+50.820768473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls") pod "image-registry-57556ff886-kmtht" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90") : secret "image-registry-tls" not found Sep 08 14:17:10.536797 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:10.536545 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls podName:c5006d65-0045-4f1a-9946-f46325714d52 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:18.536527008 +0000 UTC m=+50.820844107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-d9tcw" (UID: "c5006d65-0045-4f1a-9946-f46325714d52") : secret "cluster-monitoring-operator-tls" not found Sep 08 14:17:10.536797 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.536581 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwrg4\" (UniqueName: \"kubernetes.io/projected/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-api-access-zwrg4\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.565784 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.565760 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:17:10.566239 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.566225 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/0.log" Sep 08 14:17:10.566276 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.566258 2578 generic.go:356] "Generic (PLEG): container finished" podID="7facf738-68e8-45a4-b6a1-6758b93a47ed" containerID="a0e521bd6a7dc20bd6eec6f841d7a47af81c4e3d7a567a8af70691d42950e5a6" exitCode=255 Sep 08 14:17:10.566379 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.566356 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" event={"ID":"7facf738-68e8-45a4-b6a1-6758b93a47ed","Type":"ContainerDied","Data":"a0e521bd6a7dc20bd6eec6f841d7a47af81c4e3d7a567a8af70691d42950e5a6"} Sep 08 14:17:10.566425 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.566412 2578 scope.go:117] "RemoveContainer" containerID="dd3ee329d5a27992ca67c2121f65fbfca367acb2dc1b04d34a9825f2220162bb" Sep 08 14:17:10.566577 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.566557 2578 scope.go:117] "RemoveContainer" containerID="a0e521bd6a7dc20bd6eec6f841d7a47af81c4e3d7a567a8af70691d42950e5a6" Sep 08 14:17:10.566772 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:10.566754 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-tbsgb_openshift-console-operator(7facf738-68e8-45a4-b6a1-6758b93a47ed)\"" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" podUID="7facf738-68e8-45a4-b6a1-6758b93a47ed" Sep 08 14:17:10.637396 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.637341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/91526aea-a23b-4e12-a739-4a4b9c73433d-crio-socket\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.637396 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.637378 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwrg4\" (UniqueName: \"kubernetes.io/projected/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-api-access-zwrg4\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.637537 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.637402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/91526aea-a23b-4e12-a739-4a4b9c73433d-data-volume\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.637537 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.637447 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.637537 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.637473 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.637676 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.637552 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/91526aea-a23b-4e12-a739-4a4b9c73433d-crio-socket\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.637676 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:10.637571 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 14:17:10.637676 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:10.637610 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 14:17:10.637676 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:10.637639 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm podName:91526aea-a23b-4e12-a739-4a4b9c73433d nodeName:}" failed. No retries permitted until 2026-09-08 14:17:11.137621984 +0000 UTC m=+43.421939077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8dx7" (UID: "91526aea-a23b-4e12-a739-4a4b9c73433d") : configmap "kube-rbac-proxy" not found Sep 08 14:17:10.637676 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:10.637668 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls podName:91526aea-a23b-4e12-a739-4a4b9c73433d nodeName:}" failed. No retries permitted until 2026-09-08 14:17:11.137650608 +0000 UTC m=+43.421967702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8dx7" (UID: "91526aea-a23b-4e12-a739-4a4b9c73433d") : secret "insights-runtime-extractor-tls" not found Sep 08 14:17:10.637882 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.637765 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/91526aea-a23b-4e12-a739-4a4b9c73433d-data-volume\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.648243 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.648218 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwrg4\" (UniqueName: \"kubernetes.io/projected/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-api-access-zwrg4\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:10.738838 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:10.738804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2x4fm\" (UID: \"9c8ce093-b801-4390-b647-cd3ea9430ab4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:10.739032 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:10.739014 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 14:17:10.739099 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:10.739074 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert podName:9c8ce093-b801-4390-b647-cd3ea9430ab4 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:18.739056566 +0000 UTC m=+51.023373679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-2x4fm" (UID: "9c8ce093-b801-4390-b647-cd3ea9430ab4") : secret "networking-console-plugin-cert" not found Sep 08 14:17:11.142894 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.142858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:11.143078 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.142910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:11.143078 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:11.143015 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 14:17:11.143167 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:11.143084 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls podName:91526aea-a23b-4e12-a739-4a4b9c73433d nodeName:}" failed. No retries permitted until 2026-09-08 14:17:12.143069801 +0000 UTC m=+44.427386894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8dx7" (UID: "91526aea-a23b-4e12-a739-4a4b9c73433d") : secret "insights-runtime-extractor-tls" not found Sep 08 14:17:11.143167 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:11.143018 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 14:17:11.143167 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:11.143132 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm podName:91526aea-a23b-4e12-a739-4a4b9c73433d nodeName:}" failed. No retries permitted until 2026-09-08 14:17:12.143123502 +0000 UTC m=+44.427440610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8dx7" (UID: "91526aea-a23b-4e12-a739-4a4b9c73433d") : configmap "kube-rbac-proxy" not found Sep 08 14:17:11.222304 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.220418 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wxm5l"] Sep 08 14:17:11.234785 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.234753 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wxm5l"] Sep 08 14:17:11.234940 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.234861 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wxm5l" Sep 08 14:17:11.237426 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.237397 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 08 14:17:11.237426 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.237415 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 08 14:17:11.237791 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.237761 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7lc4b\"" Sep 08 14:17:11.344644 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.344605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjgxc\" (UniqueName: \"kubernetes.io/projected/88047337-66c2-44bb-8f97-3a98a11cc754-kube-api-access-sjgxc\") pod \"migrator-fdbb8fdff-wxm5l\" (UID: \"88047337-66c2-44bb-8f97-3a98a11cc754\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wxm5l" Sep 08 14:17:11.445479 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.445410 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjgxc\" (UniqueName: \"kubernetes.io/projected/88047337-66c2-44bb-8f97-3a98a11cc754-kube-api-access-sjgxc\") pod \"migrator-fdbb8fdff-wxm5l\" (UID: \"88047337-66c2-44bb-8f97-3a98a11cc754\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wxm5l" Sep 08 14:17:11.458040 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.458018 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjgxc\" (UniqueName: \"kubernetes.io/projected/88047337-66c2-44bb-8f97-3a98a11cc754-kube-api-access-sjgxc\") pod \"migrator-fdbb8fdff-wxm5l\" (UID: \"88047337-66c2-44bb-8f97-3a98a11cc754\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wxm5l" Sep 08 14:17:11.543329 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.543303 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wxm5l" Sep 08 14:17:11.570424 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.570405 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:17:11.570720 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.570708 2578 scope.go:117] "RemoveContainer" containerID="a0e521bd6a7dc20bd6eec6f841d7a47af81c4e3d7a567a8af70691d42950e5a6" Sep 08 14:17:11.570885 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:11.570864 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-tbsgb_openshift-console-operator(7facf738-68e8-45a4-b6a1-6758b93a47ed)\"" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" podUID="7facf738-68e8-45a4-b6a1-6758b93a47ed" Sep 08 14:17:11.832678 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:11.832647 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wxm5l"] Sep 08 14:17:11.835421 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:11.835394 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88047337_66c2_44bb_8f97_3a98a11cc754.slice/crio-e1ace4219b8a5d64b10688d18c17f9b9537d205df778d12f92e5bb73aadc487d WatchSource:0}: Error finding container e1ace4219b8a5d64b10688d18c17f9b9537d205df778d12f92e5bb73aadc487d: Status 404 returned error can't find the container with id e1ace4219b8a5d64b10688d18c17f9b9537d205df778d12f92e5bb73aadc487d Sep 08 14:17:12.145926 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:12.145860 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xrx9d_f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c/dns-node-resolver/0.log" Sep 08 14:17:12.152219 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:12.152192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:12.152334 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:12.152230 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:12.152334 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:12.152328 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 14:17:12.152428 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:12.152353 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 14:17:12.152428 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:12.152383 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm podName:91526aea-a23b-4e12-a739-4a4b9c73433d nodeName:}" failed. No retries permitted until 2026-09-08 14:17:14.152366288 +0000 UTC m=+46.436683382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8dx7" (UID: "91526aea-a23b-4e12-a739-4a4b9c73433d") : configmap "kube-rbac-proxy" not found Sep 08 14:17:12.152428 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:12.152405 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls podName:91526aea-a23b-4e12-a739-4a4b9c73433d nodeName:}" failed. No retries permitted until 2026-09-08 14:17:14.152391557 +0000 UTC m=+46.436708650 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8dx7" (UID: "91526aea-a23b-4e12-a739-4a4b9c73433d") : secret "insights-runtime-extractor-tls" not found Sep 08 14:17:12.575363 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:12.575324 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wxm5l" event={"ID":"88047337-66c2-44bb-8f97-3a98a11cc754","Type":"ContainerStarted","Data":"e1ace4219b8a5d64b10688d18c17f9b9537d205df778d12f92e5bb73aadc487d"} Sep 08 14:17:13.074679 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:13.074639 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:13.074864 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:13.074712 2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:13.075160 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:13.075146 2578 scope.go:117] "RemoveContainer" containerID="a0e521bd6a7dc20bd6eec6f841d7a47af81c4e3d7a567a8af70691d42950e5a6" Sep 08 14:17:13.075355 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:13.075333 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-tbsgb_openshift-console-operator(7facf738-68e8-45a4-b6a1-6758b93a47ed)\"" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" podUID="7facf738-68e8-45a4-b6a1-6758b93a47ed" Sep 08 14:17:13.152337 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:13.152297 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hlpbr_a4d496bc-aa63-48dd-b346-e04b0468e03a/node-ca/0.log" Sep 08 14:17:13.746650 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:13.746623 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pxs2b_38343840-71de-4d89-b76f-d628180edba4/serve-healthcheck-canary/0.log" Sep 08 14:17:14.169105 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:14.169073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:14.169293 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:14.169120 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:14.169293 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:14.169199 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 08 14:17:14.169293 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:14.169247 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 14:17:14.169422 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:14.169251 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm podName:91526aea-a23b-4e12-a739-4a4b9c73433d nodeName:}" failed. No retries permitted until 2026-09-08 14:17:18.169238422 +0000 UTC m=+50.453555514 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8dx7" (UID: "91526aea-a23b-4e12-a739-4a4b9c73433d") : configmap "kube-rbac-proxy" not found Sep 08 14:17:14.169422 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:14.169322 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls podName:91526aea-a23b-4e12-a739-4a4b9c73433d nodeName:}" failed. No retries permitted until 2026-09-08 14:17:18.169304837 +0000 UTC m=+50.453621931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8dx7" (UID: "91526aea-a23b-4e12-a739-4a4b9c73433d") : secret "insights-runtime-extractor-tls" not found Sep 08 14:17:14.582776 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:14.582744 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wxm5l" event={"ID":"88047337-66c2-44bb-8f97-3a98a11cc754","Type":"ContainerStarted","Data":"08122a3dfab1418bdda82de0ca71f980b121ae2ab3ff2018ea1c7fe5465f6810"} Sep 08 14:17:14.582933 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:14.582793 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wxm5l" event={"ID":"88047337-66c2-44bb-8f97-3a98a11cc754","Type":"ContainerStarted","Data":"3708365dad30d232b87fbefc5cd639159f067073eb5798e8acd65c1b7289aef5"} Sep 08 14:17:14.598934 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:14.598872 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-wxm5l" podStartSLOduration=1.823308838 podStartE2EDuration="3.598854479s" totalImagesPullingTime="1.775545641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:14.598075777 +0000 UTC m=+46.882392892" watchObservedRunningTime="2026-09-08 14:17:14.598854479 +0000 UTC m=+46.883171594" Sep 08 14:17:17.900400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:17.900364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:17.900824 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:17.900513 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 08 14:17:17.900824 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:17.900581 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls podName:fa59392d-7939-41be-bcb6-0a382dfa8583 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:33.900565131 +0000 UTC m=+66.184882228 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls") pod "dns-default-4wnjb" (UID: "fa59392d-7939-41be-bcb6-0a382dfa8583") : secret "dns-default-metrics-tls" not found Sep 08 14:17:18.203864 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:18.203785 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:18.203864 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:18.203842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:18.204020 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:18.203928 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 08 14:17:18.204020 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:18.203986 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls podName:91526aea-a23b-4e12-a739-4a4b9c73433d nodeName:}" failed. No retries permitted until 2026-09-08 14:17:26.203971803 +0000 UTC m=+58.488288896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8dx7" (UID: "91526aea-a23b-4e12-a739-4a4b9c73433d") : secret "insights-runtime-extractor-tls" not found Sep 08 14:17:18.204277 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:18.204261 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/91526aea-a23b-4e12-a739-4a4b9c73433d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:18.607014 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:18.606984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:18.607174 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:18.607083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:18.607174 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:18.607129 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 08 14:17:18.607244 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:18.607183 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 08 14:17:18.607244 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:18.607195 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57556ff886-kmtht: secret "image-registry-tls" not found Sep 08 14:17:18.607244 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:18.607199 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls podName:c5006d65-0045-4f1a-9946-f46325714d52 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:34.607184352 +0000 UTC m=+66.891501444 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-d9tcw" (UID: "c5006d65-0045-4f1a-9946-f46325714d52") : secret "cluster-monitoring-operator-tls" not found Sep 08 14:17:18.607244 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:18.607231 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls podName:b31922ec-e476-4ca8-9d58-e64859608a90 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:34.607219196 +0000 UTC m=+66.891536288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls") pod "image-registry-57556ff886-kmtht" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90") : secret "image-registry-tls" not found Sep 08 14:17:18.808946 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:18.808912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2x4fm\" (UID: \"9c8ce093-b801-4390-b647-cd3ea9430ab4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:18.809096 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:18.809070 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 08 14:17:18.809155 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:18.809139 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert podName:9c8ce093-b801-4390-b647-cd3ea9430ab4 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:34.809123831 +0000 UTC m=+67.093440924 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-2x4fm" (UID: "9c8ce093-b801-4390-b647-cd3ea9430ab4") : secret "networking-console-plugin-cert" not found Sep 08 14:17:26.264939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:26.264909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:26.267217 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:26.267199 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/91526aea-a23b-4e12-a739-4a4b9c73433d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8dx7\" (UID: \"91526aea-a23b-4e12-a739-4a4b9c73433d\") " pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:26.311711 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:26.311682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8dx7" Sep 08 14:17:26.534159 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:26.534089 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tlbbx" Sep 08 14:17:26.611007 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:26.610979 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8dx7"] Sep 08 14:17:26.614192 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:26.614167 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91526aea_a23b_4e12_a739_4a4b9c73433d.slice/crio-b7bb8cb60f9651179b4d3360ddea4db89363403262e396d9c3505fbfb846e301 WatchSource:0}: Error finding container b7bb8cb60f9651179b4d3360ddea4db89363403262e396d9c3505fbfb846e301: Status 404 returned error can't find the container with id b7bb8cb60f9651179b4d3360ddea4db89363403262e396d9c3505fbfb846e301 Sep 08 14:17:26.619085 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:26.619060 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8dx7" event={"ID":"91526aea-a23b-4e12-a739-4a4b9c73433d","Type":"ContainerStarted","Data":"b7bb8cb60f9651179b4d3360ddea4db89363403262e396d9c3505fbfb846e301"} Sep 08 14:17:27.624069 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:27.624037 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8dx7" event={"ID":"91526aea-a23b-4e12-a739-4a4b9c73433d","Type":"ContainerStarted","Data":"a337caa8d498b24a0a242f77fdb7e529d34792aedcc223005350e40d6606d9c9"} Sep 08 14:17:28.236863 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:28.236829 2578 scope.go:117] "RemoveContainer" containerID="a0e521bd6a7dc20bd6eec6f841d7a47af81c4e3d7a567a8af70691d42950e5a6" Sep 08 14:17:28.629706 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:28.629627 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:17:28.630144 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:28.629767 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" event={"ID":"7facf738-68e8-45a4-b6a1-6758b93a47ed","Type":"ContainerStarted","Data":"003aaffa12c6e5f5313bd14ac2ad7802a821ed80e1f947be6694f9448212275e"} Sep 08 14:17:28.630439 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:28.630418 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:28.632040 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:28.632013 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8dx7" event={"ID":"91526aea-a23b-4e12-a739-4a4b9c73433d","Type":"ContainerStarted","Data":"0051b57c3460bbc08d871fcc4e22f3380a4b29b0fa5f36f50416aa2b149974bf"} Sep 08 14:17:28.648488 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:28.648450 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" podStartSLOduration=22.085457282 podStartE2EDuration="26.64843689s" totalImagesPullingTime="4.562979608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:28.647926881 +0000 UTC m=+60.932243997" watchObservedRunningTime="2026-09-08 14:17:28.64843689 +0000 UTC m=+60.932754004" Sep 08 14:17:29.630978 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:29.630936 2578 patch_prober.go:28] interesting pod/console-operator-85b454874b-tbsgb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.133.0.8:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 08 14:17:29.631359 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:29.631001 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" podUID="7facf738-68e8-45a4-b6a1-6758b93a47ed" containerName="console-operator" probeResult="failure" output="Get \"https://10.133.0.8:8443/readyz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 08 14:17:29.638439 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:29.638394 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8dx7" event={"ID":"91526aea-a23b-4e12-a739-4a4b9c73433d","Type":"ContainerStarted","Data":"322358e839c6b8d6dc3926973443dfae511e9acac9d060c2382666ea67cd2179"} Sep 08 14:17:29.642442 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:29.642421 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-tbsgb" Sep 08 14:17:29.657099 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:29.657036 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f8dx7" podStartSLOduration=17.306582902 podStartE2EDuration="19.657016994s" totalImagesPullingTime="2.350434092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:17:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:29.656675736 +0000 UTC m=+61.940992852" watchObservedRunningTime="2026-09-08 14:17:29.657016994 +0000 UTC m=+61.941334110" Sep 08 14:17:32.922543 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:32.922502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:17:32.923070 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:32.922568 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs\") pod \"network-metrics-daemon-998rf\" (UID: \"5267f2dc-393e-40f2-8b78-25179961c1a3\") " pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:17:32.925109 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:32.925090 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 08 14:17:32.925191 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:32.925090 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 08 14:17:32.935011 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:32.934992 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b4afcd3a-ef6b-4488-89b7-c48529821f6c-original-pull-secret\") pod \"global-pull-secret-syncer-hsh78\" (UID: \"b4afcd3a-ef6b-4488-89b7-c48529821f6c\") " pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:17:32.935095 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:32.935042 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267f2dc-393e-40f2-8b78-25179961c1a3-metrics-certs\") pod \"network-metrics-daemon-998rf\" (UID: \"5267f2dc-393e-40f2-8b78-25179961c1a3\") " pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:17:32.946080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:32.946062 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsh78" Sep 08 14:17:32.954125 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:32.954111 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zbnn7\"" Sep 08 14:17:32.962808 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:32.962786 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-998rf" Sep 08 14:17:32.985216 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:32.985190 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57556ff886-kmtht"] Sep 08 14:17:32.985436 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:32.985414 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-57556ff886-kmtht" podUID="b31922ec-e476-4ca8-9d58-e64859608a90" Sep 08 14:17:33.024055 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.023927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8d5w\" (UniqueName: \"kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w\") pod \"network-check-target-4njj8\" (UID: \"43ad5ec2-69c5-4c3a-a997-adb20472335a\") " pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:17:33.026484 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.026416 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 08 14:17:33.036389 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.036367 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 08 14:17:33.047385 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.047363 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8d5w\" (UniqueName: \"kubernetes.io/projected/43ad5ec2-69c5-4c3a-a997-adb20472335a-kube-api-access-f8d5w\") pod \"network-check-target-4njj8\" (UID: \"43ad5ec2-69c5-4c3a-a997-adb20472335a\") " pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:17:33.099584 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.099554 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77bc4bb45d-c2pxl"] Sep 08 14:17:33.113586 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.113563 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.117845 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.117823 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77bc4bb45d-c2pxl"] Sep 08 14:17:33.147852 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.147830 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vvt7d\"" Sep 08 14:17:33.155990 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.155967 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:17:33.219337 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.219309 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hsh78"] Sep 08 14:17:33.223204 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:33.223182 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4afcd3a_ef6b_4488_89b7_c48529821f6c.slice/crio-4d6e67f92d5de97425105d06992e3aa03fd6d2511897a4483def3ee6c7b48dce WatchSource:0}: Error finding container 4d6e67f92d5de97425105d06992e3aa03fd6d2511897a4483def3ee6c7b48dce: Status 404 returned error can't find the container with id 4d6e67f92d5de97425105d06992e3aa03fd6d2511897a4483def3ee6c7b48dce Sep 08 14:17:33.225192 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.225170 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdtp5\" (UniqueName: \"kubernetes.io/projected/b188798f-94d5-4067-9247-8e18472be130-kube-api-access-xdtp5\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.225267 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.225250 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b188798f-94d5-4067-9247-8e18472be130-installation-pull-secrets\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.225348 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.225334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b188798f-94d5-4067-9247-8e18472be130-image-registry-private-configuration\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.225405 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.225363 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b188798f-94d5-4067-9247-8e18472be130-ca-trust-extracted\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.225405 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.225399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b188798f-94d5-4067-9247-8e18472be130-registry-tls\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.225491 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.225422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b188798f-94d5-4067-9247-8e18472be130-bound-sa-token\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.225524 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.225491 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b188798f-94d5-4067-9247-8e18472be130-registry-certificates\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.225592 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.225541 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b188798f-94d5-4067-9247-8e18472be130-trusted-ca\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.246575 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.246551 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-998rf"] Sep 08 14:17:33.249356 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:33.249335 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5267f2dc_393e_40f2_8b78_25179961c1a3.slice/crio-e236a84c6b3826da66198fc92a04967828eacf4e0dec30f58f0565376ca1eae7 WatchSource:0}: Error finding container e236a84c6b3826da66198fc92a04967828eacf4e0dec30f58f0565376ca1eae7: Status 404 returned error can't find the container with id e236a84c6b3826da66198fc92a04967828eacf4e0dec30f58f0565376ca1eae7 Sep 08 14:17:33.326809 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.326773 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdtp5\" (UniqueName: \"kubernetes.io/projected/b188798f-94d5-4067-9247-8e18472be130-kube-api-access-xdtp5\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.326967 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.326815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b188798f-94d5-4067-9247-8e18472be130-installation-pull-secrets\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.327040 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.327018 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b188798f-94d5-4067-9247-8e18472be130-image-registry-private-configuration\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.327097 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.327050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b188798f-94d5-4067-9247-8e18472be130-ca-trust-extracted\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.327097 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.327071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b188798f-94d5-4067-9247-8e18472be130-registry-tls\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.327097 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.327089 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b188798f-94d5-4067-9247-8e18472be130-bound-sa-token\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.327267 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.327239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b188798f-94d5-4067-9247-8e18472be130-registry-certificates\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.327346 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.327327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b188798f-94d5-4067-9247-8e18472be130-trusted-ca\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.327445 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.327423 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b188798f-94d5-4067-9247-8e18472be130-ca-trust-extracted\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.327958 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.327937 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b188798f-94d5-4067-9247-8e18472be130-registry-certificates\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.328188 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.328171 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b188798f-94d5-4067-9247-8e18472be130-trusted-ca\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.329464 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.329445 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b188798f-94d5-4067-9247-8e18472be130-registry-tls\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.330075 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.330054 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b188798f-94d5-4067-9247-8e18472be130-image-registry-private-configuration\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.330159 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.330116 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b188798f-94d5-4067-9247-8e18472be130-installation-pull-secrets\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.334870 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.334851 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b188798f-94d5-4067-9247-8e18472be130-bound-sa-token\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.335165 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.335146 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdtp5\" (UniqueName: \"kubernetes.io/projected/b188798f-94d5-4067-9247-8e18472be130-kube-api-access-xdtp5\") pod \"image-registry-77bc4bb45d-c2pxl\" (UID: \"b188798f-94d5-4067-9247-8e18472be130\") " pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.426818 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.426788 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kj4kg\"" Sep 08 14:17:33.433961 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.433922 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:33.435227 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.435201 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4njj8"] Sep 08 14:17:33.439045 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:33.439024 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ad5ec2_69c5_4c3a_a997_adb20472335a.slice/crio-51ff7f9e687b47e5650920aa9d617a15bdd06cd0b3ba9b9707330e10980c6641 WatchSource:0}: Error finding container 51ff7f9e687b47e5650920aa9d617a15bdd06cd0b3ba9b9707330e10980c6641: Status 404 returned error can't find the container with id 51ff7f9e687b47e5650920aa9d617a15bdd06cd0b3ba9b9707330e10980c6641 Sep 08 14:17:33.652502 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.652462 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-998rf" event={"ID":"5267f2dc-393e-40f2-8b78-25179961c1a3","Type":"ContainerStarted","Data":"e236a84c6b3826da66198fc92a04967828eacf4e0dec30f58f0565376ca1eae7"} Sep 08 14:17:33.653706 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.653676 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4njj8" event={"ID":"43ad5ec2-69c5-4c3a-a997-adb20472335a","Type":"ContainerStarted","Data":"51ff7f9e687b47e5650920aa9d617a15bdd06cd0b3ba9b9707330e10980c6641"} Sep 08 14:17:33.654843 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.654825 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:33.654955 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.654852 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hsh78" event={"ID":"b4afcd3a-ef6b-4488-89b7-c48529821f6c","Type":"ContainerStarted","Data":"4d6e67f92d5de97425105d06992e3aa03fd6d2511897a4483def3ee6c7b48dce"} Sep 08 14:17:33.660446 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.660430 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:33.706565 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.706538 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77bc4bb45d-c2pxl"] Sep 08 14:17:33.710433 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:33.710404 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb188798f_94d5_4067_9247_8e18472be130.slice/crio-1dde928369c5e6714e9a41248001503238e07a934183c0f657ae5fe36012e257 WatchSource:0}: Error finding container 1dde928369c5e6714e9a41248001503238e07a934183c0f657ae5fe36012e257: Status 404 returned error can't find the container with id 1dde928369c5e6714e9a41248001503238e07a934183c0f657ae5fe36012e257 Sep 08 14:17:33.730311 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.730259 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b31922ec-e476-4ca8-9d58-e64859608a90-image-registry-private-configuration\") pod \"b31922ec-e476-4ca8-9d58-e64859608a90\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " Sep 08 14:17:33.730407 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.730306 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-bound-sa-token\") pod \"b31922ec-e476-4ca8-9d58-e64859608a90\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " Sep 08 14:17:33.730407 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.730347 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b31922ec-e476-4ca8-9d58-e64859608a90-ca-trust-extracted\") pod \"b31922ec-e476-4ca8-9d58-e64859608a90\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " Sep 08 14:17:33.730407 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.730387 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b31922ec-e476-4ca8-9d58-e64859608a90-registry-certificates\") pod \"b31922ec-e476-4ca8-9d58-e64859608a90\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " Sep 08 14:17:33.730642 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.730418 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b31922ec-e476-4ca8-9d58-e64859608a90-installation-pull-secrets\") pod \"b31922ec-e476-4ca8-9d58-e64859608a90\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " Sep 08 14:17:33.730642 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.730451 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2qmt\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-kube-api-access-t2qmt\") pod \"b31922ec-e476-4ca8-9d58-e64859608a90\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " Sep 08 14:17:33.730642 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.730500 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b31922ec-e476-4ca8-9d58-e64859608a90-trusted-ca\") pod \"b31922ec-e476-4ca8-9d58-e64859608a90\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " Sep 08 14:17:33.731079 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.730857 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31922ec-e476-4ca8-9d58-e64859608a90-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b31922ec-e476-4ca8-9d58-e64859608a90" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:17:33.731226 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.731200 2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b31922ec-e476-4ca8-9d58-e64859608a90-registry-certificates\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:17:33.731907 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.731877 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31922ec-e476-4ca8-9d58-e64859608a90-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b31922ec-e476-4ca8-9d58-e64859608a90" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:17:33.731999 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.731932 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31922ec-e476-4ca8-9d58-e64859608a90-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b31922ec-e476-4ca8-9d58-e64859608a90" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:17:33.732793 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.732769 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31922ec-e476-4ca8-9d58-e64859608a90-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b31922ec-e476-4ca8-9d58-e64859608a90" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:17:33.733935 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.733825 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b31922ec-e476-4ca8-9d58-e64859608a90" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:17:33.733935 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.733893 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31922ec-e476-4ca8-9d58-e64859608a90-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b31922ec-e476-4ca8-9d58-e64859608a90" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:17:33.734051 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.733993 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-kube-api-access-t2qmt" (OuterVolumeSpecName: "kube-api-access-t2qmt") pod "b31922ec-e476-4ca8-9d58-e64859608a90" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90"). InnerVolumeSpecName "kube-api-access-t2qmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:17:33.832172 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.832134 2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b31922ec-e476-4ca8-9d58-e64859608a90-installation-pull-secrets\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:17:33.832172 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.832174 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qmt\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-kube-api-access-t2qmt\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:17:33.832406 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.832190 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b31922ec-e476-4ca8-9d58-e64859608a90-trusted-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:17:33.832406 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.832205 2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b31922ec-e476-4ca8-9d58-e64859608a90-image-registry-private-configuration\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:17:33.832406 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.832221 2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-bound-sa-token\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:17:33.832406 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.832234 2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b31922ec-e476-4ca8-9d58-e64859608a90-ca-trust-extracted\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:17:33.933419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.933376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:33.936860 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:33.936833 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fa59392d-7939-41be-bcb6-0a382dfa8583-metrics-tls\") pod \"dns-default-4wnjb\" (UID: \"fa59392d-7939-41be-bcb6-0a382dfa8583\") " pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:34.157309 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.157224 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rlxm6\"" Sep 08 14:17:34.166297 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.166012 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:34.590697 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.590632 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4wnjb"] Sep 08 14:17:34.599287 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:34.599254 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa59392d_7939_41be_bcb6_0a382dfa8583.slice/crio-eb3d08c325cec624b093a1ba2eb1a4ff011797b069e3106d53d254e0c8e40cad WatchSource:0}: Error finding container eb3d08c325cec624b093a1ba2eb1a4ff011797b069e3106d53d254e0c8e40cad: Status 404 returned error can't find the container with id eb3d08c325cec624b093a1ba2eb1a4ff011797b069e3106d53d254e0c8e40cad Sep 08 14:17:34.640490 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.640458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:34.640626 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.640546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:34.643615 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.643567 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c5006d65-0045-4f1a-9946-f46325714d52-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-d9tcw\" (UID: \"c5006d65-0045-4f1a-9946-f46325714d52\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:34.644127 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.644103 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls\") pod \"image-registry-57556ff886-kmtht\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:34.663916 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.663885 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" event={"ID":"b188798f-94d5-4067-9247-8e18472be130","Type":"ContainerStarted","Data":"dcbe644351f6ec93bacf1a08fc16be4c0ae0ad59f73c4c8accefafe4fcbad83e"} Sep 08 14:17:34.664030 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.663986 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" event={"ID":"b188798f-94d5-4067-9247-8e18472be130","Type":"ContainerStarted","Data":"1dde928369c5e6714e9a41248001503238e07a934183c0f657ae5fe36012e257"} Sep 08 14:17:34.664490 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.664465 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:34.668076 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.668048 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-998rf" event={"ID":"5267f2dc-393e-40f2-8b78-25179961c1a3","Type":"ContainerStarted","Data":"34b26935f012802b6a92a2965bff61d80525b591202725a20c43dfddaaf5878c"} Sep 08 14:17:34.668185 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.668099 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-998rf" event={"ID":"5267f2dc-393e-40f2-8b78-25179961c1a3","Type":"ContainerStarted","Data":"2656e3d2c4f38fa7848d2a8ecf6a3dd514d5933320dcecc719fa42b20d6787e5"} Sep 08 14:17:34.670792 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.670768 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57556ff886-kmtht" Sep 08 14:17:34.670904 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.670876 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4wnjb" event={"ID":"fa59392d-7939-41be-bcb6-0a382dfa8583","Type":"ContainerStarted","Data":"eb3d08c325cec624b093a1ba2eb1a4ff011797b069e3106d53d254e0c8e40cad"} Sep 08 14:17:34.709804 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.709750 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" podStartSLOduration=1.709708239 podStartE2EDuration="1.709708239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:34.690917289 +0000 UTC m=+66.975234405" watchObservedRunningTime="2026-09-08 14:17:34.709708239 +0000 UTC m=+66.994025360" Sep 08 14:17:34.710487 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.710450 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-998rf" podStartSLOduration=65.623461456 podStartE2EDuration="1m6.710437147s" totalImagesPullingTime="1.086975691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:34.710064602 +0000 UTC m=+66.994381719" watchObservedRunningTime="2026-09-08 14:17:34.710437147 +0000 UTC m=+66.994754262" Sep 08 14:17:34.740987 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.740961 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls\") pod \"b31922ec-e476-4ca8-9d58-e64859608a90\" (UID: \"b31922ec-e476-4ca8-9d58-e64859608a90\") " Sep 08 14:17:34.744171 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.744134 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b31922ec-e476-4ca8-9d58-e64859608a90" (UID: "b31922ec-e476-4ca8-9d58-e64859608a90"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:17:34.842699 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.842661 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2x4fm\" (UID: \"9c8ce093-b801-4390-b647-cd3ea9430ab4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:34.842878 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.842749 2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31922ec-e476-4ca8-9d58-e64859608a90-registry-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:17:34.845376 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.845349 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9c8ce093-b801-4390-b647-cd3ea9430ab4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-2x4fm\" (UID: \"9c8ce093-b801-4390-b647-cd3ea9430ab4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:34.885311 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.885282 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m9ct5\"" Sep 08 14:17:34.894130 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:34.894111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" Sep 08 14:17:35.012946 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:35.012868 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57556ff886-kmtht"] Sep 08 14:17:35.021786 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:35.021761 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57556ff886-kmtht"] Sep 08 14:17:35.080962 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:35.080930 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-kkhk5\"" Sep 08 14:17:35.089699 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:35.089661 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" Sep 08 14:17:35.182162 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:35.182062 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw"] Sep 08 14:17:35.188593 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:35.188560 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5006d65_0045_4f1a_9946_f46325714d52.slice/crio-e6cb569c9a5d0e555dfc08474dd7caafaca69edac03afce918c2933a5c8400ae WatchSource:0}: Error finding container e6cb569c9a5d0e555dfc08474dd7caafaca69edac03afce918c2933a5c8400ae: Status 404 returned error can't find the container with id e6cb569c9a5d0e555dfc08474dd7caafaca69edac03afce918c2933a5c8400ae Sep 08 14:17:35.383692 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:35.383604 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-2x4fm"] Sep 08 14:17:35.388226 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:35.388194 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8ce093_b801_4390_b647_cd3ea9430ab4.slice/crio-53633d4fdc30c4e859b17274fd09a467421a5e6799f87005e35ce050fa893b9c WatchSource:0}: Error finding container 53633d4fdc30c4e859b17274fd09a467421a5e6799f87005e35ce050fa893b9c: Status 404 returned error can't find the container with id 53633d4fdc30c4e859b17274fd09a467421a5e6799f87005e35ce050fa893b9c Sep 08 14:17:35.678129 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:35.678004 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" event={"ID":"9c8ce093-b801-4390-b647-cd3ea9430ab4","Type":"ContainerStarted","Data":"53633d4fdc30c4e859b17274fd09a467421a5e6799f87005e35ce050fa893b9c"} Sep 08 14:17:35.680405 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:35.680367 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" event={"ID":"c5006d65-0045-4f1a-9946-f46325714d52","Type":"ContainerStarted","Data":"e6cb569c9a5d0e555dfc08474dd7caafaca69edac03afce918c2933a5c8400ae"} Sep 08 14:17:36.239947 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:36.239901 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b31922ec-e476-4ca8-9d58-e64859608a90" path="/var/lib/kubelet/pods/b31922ec-e476-4ca8-9d58-e64859608a90/volumes" Sep 08 14:17:39.703754 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:39.703694 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4njj8" event={"ID":"43ad5ec2-69c5-4c3a-a997-adb20472335a","Type":"ContainerStarted","Data":"b051bc6c0d3d8542a43d100b7f2c0561ea2442d4c11ebf0d9a1bb981b79f67cb"} Sep 08 14:17:39.704223 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:39.703902 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:17:39.705700 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:39.705673 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hsh78" event={"ID":"b4afcd3a-ef6b-4488-89b7-c48529821f6c","Type":"ContainerStarted","Data":"a4b5ea91ee1d195533ebe9fad72e25774a774ec383271786e2c5fe8afeae6d4d"} Sep 08 14:17:39.707818 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:39.707793 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4wnjb" event={"ID":"fa59392d-7939-41be-bcb6-0a382dfa8583","Type":"ContainerStarted","Data":"75fa419e188d52c6af201334f1324ca84f4ae77e6d7afdaf1f12c3e4d63ae8aa"} Sep 08 14:17:39.707917 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:39.707841 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4wnjb" event={"ID":"fa59392d-7939-41be-bcb6-0a382dfa8583","Type":"ContainerStarted","Data":"39e72e0dba5b84beb9eb3c7c9435f1fed5db14b49baf5f6b9e9140ba936d4be4"} Sep 08 14:17:39.707973 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:39.707945 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:39.718856 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:39.718783 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4njj8" podStartSLOduration=66.3522642 podStartE2EDuration="1m11.71876286s" totalImagesPullingTime="5.36649866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:39.718384875 +0000 UTC m=+72.002701990" watchObservedRunningTime="2026-09-08 14:17:39.71876286 +0000 UTC m=+72.003079972" Sep 08 14:17:39.740170 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:39.740123 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hsh78" podStartSLOduration=66.152407807 podStartE2EDuration="1m11.7401088s" totalImagesPullingTime="5.587700993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:39.739254631 +0000 UTC m=+72.023571747" watchObservedRunningTime="2026-09-08 14:17:39.7401088 +0000 UTC m=+72.024425907" Sep 08 14:17:39.758655 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:39.758617 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4wnjb" podStartSLOduration=33.561945402 podStartE2EDuration="37.758601345s" totalImagesPullingTime="4.196655943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:39.758048599 +0000 UTC m=+72.042365714" watchObservedRunningTime="2026-09-08 14:17:39.758601345 +0000 UTC m=+72.042918461" Sep 08 14:17:41.494022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:41.493987 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx"] Sep 08 14:17:41.513365 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:41.513343 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx"] Sep 08 14:17:41.513492 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:41.513428 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx" Sep 08 14:17:41.515415 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:41.515397 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 08 14:17:41.515516 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:41.515497 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pxhc6\"" Sep 08 14:17:41.596054 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:41.596027 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a4b296eb-06db-4b56-8a44-8d4efea6e336-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zkqdx\" (UID: \"a4b296eb-06db-4b56-8a44-8d4efea6e336\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx" Sep 08 14:17:41.697433 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:41.697399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a4b296eb-06db-4b56-8a44-8d4efea6e336-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zkqdx\" (UID: \"a4b296eb-06db-4b56-8a44-8d4efea6e336\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx" Sep 08 14:17:41.697568 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:41.697543 2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 08 14:17:41.697626 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:41.697617 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4b296eb-06db-4b56-8a44-8d4efea6e336-tls-certificates podName:a4b296eb-06db-4b56-8a44-8d4efea6e336 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:42.197596461 +0000 UTC m=+74.481913555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a4b296eb-06db-4b56-8a44-8d4efea6e336-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-zkqdx" (UID: "a4b296eb-06db-4b56-8a44-8d4efea6e336") : secret "prometheus-operator-admission-webhook-tls" not found Sep 08 14:17:41.717914 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:41.717880 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" event={"ID":"9c8ce093-b801-4390-b647-cd3ea9430ab4","Type":"ContainerStarted","Data":"677f3f66cdd3876783598d59b7f0576e02ea6a7fd2af2d42bce4a3b9e95bb86a"} Sep 08 14:17:41.719403 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:41.719379 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" event={"ID":"c5006d65-0045-4f1a-9946-f46325714d52","Type":"ContainerStarted","Data":"548219edeeeb1b722f9f8887be6026aca0da5c4bd27cef6c2a1aef6216fea4ee"} Sep 08 14:17:41.736800 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:41.736758 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-2x4fm" podStartSLOduration=34.502863268 podStartE2EDuration="39.73672018s" totalImagesPullingTime="5.233856912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:41.736298633 +0000 UTC m=+74.020615748" watchObservedRunningTime="2026-09-08 14:17:41.73672018 +0000 UTC m=+74.021037295" Sep 08 14:17:41.752262 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:41.752186 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-d9tcw" podStartSLOduration=33.997342446 podStartE2EDuration="39.752170816s" totalImagesPullingTime="5.75482837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:41.751178944 +0000 UTC m=+74.035496059" watchObservedRunningTime="2026-09-08 14:17:41.752170816 +0000 UTC m=+74.036487976" Sep 08 14:17:42.200558 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:42.200475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a4b296eb-06db-4b56-8a44-8d4efea6e336-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zkqdx\" (UID: \"a4b296eb-06db-4b56-8a44-8d4efea6e336\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx" Sep 08 14:17:42.203230 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:42.203205 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a4b296eb-06db-4b56-8a44-8d4efea6e336-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-zkqdx\" (UID: \"a4b296eb-06db-4b56-8a44-8d4efea6e336\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx" Sep 08 14:17:42.422839 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:42.422797 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx" Sep 08 14:17:42.706619 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:42.706593 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx"] Sep 08 14:17:42.708935 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:42.708903 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b296eb_06db_4b56_8a44_8d4efea6e336.slice/crio-2ad006d39b6f73b3d442de1f6880d4739bc26d9babb44c488e9a73a2792f2df7 WatchSource:0}: Error finding container 2ad006d39b6f73b3d442de1f6880d4739bc26d9babb44c488e9a73a2792f2df7: Status 404 returned error can't find the container with id 2ad006d39b6f73b3d442de1f6880d4739bc26d9babb44c488e9a73a2792f2df7 Sep 08 14:17:42.723574 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:42.723544 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx" event={"ID":"a4b296eb-06db-4b56-8a44-8d4efea6e336","Type":"ContainerStarted","Data":"2ad006d39b6f73b3d442de1f6880d4739bc26d9babb44c488e9a73a2792f2df7"} Sep 08 14:17:44.732268 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:44.732230 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx" event={"ID":"a4b296eb-06db-4b56-8a44-8d4efea6e336","Type":"ContainerStarted","Data":"917d7e54d7057a25b9650ff3861b3296a1f6e1ea69a937a5cbdc88758374dbc8"} Sep 08 14:17:44.732781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:44.732401 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx" Sep 08 14:17:44.737038 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:44.737021 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx" Sep 08 14:17:44.748093 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:44.748053 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-zkqdx" podStartSLOduration=2.481354006 podStartE2EDuration="3.748040689s" totalImagesPullingTime="1.266686683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:44.746415498 +0000 UTC m=+77.030732612" watchObservedRunningTime="2026-09-08 14:17:44.748040689 +0000 UTC m=+77.032357803" Sep 08 14:17:45.555576 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.555536 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-t7blt"] Sep 08 14:17:45.557955 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.557934 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:45.560874 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.560850 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 08 14:17:45.560992 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.560889 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 08 14:17:45.560992 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.560933 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 08 14:17:45.560992 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.560974 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-86dl4\"" Sep 08 14:17:45.566477 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.566456 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-t7blt"] Sep 08 14:17:45.626598 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.626577 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b195df79-0181-4905-86b9-121fdbb67623-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-t7blt\" (UID: \"b195df79-0181-4905-86b9-121fdbb67623\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:45.626710 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.626616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b195df79-0181-4905-86b9-121fdbb67623-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-t7blt\" (UID: \"b195df79-0181-4905-86b9-121fdbb67623\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:45.626710 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.626635 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg24d\" (UniqueName: \"kubernetes.io/projected/b195df79-0181-4905-86b9-121fdbb67623-kube-api-access-mg24d\") pod \"prometheus-operator-6f4946db56-t7blt\" (UID: \"b195df79-0181-4905-86b9-121fdbb67623\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:45.626710 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.626663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b195df79-0181-4905-86b9-121fdbb67623-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-t7blt\" (UID: \"b195df79-0181-4905-86b9-121fdbb67623\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:45.727524 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.727477 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b195df79-0181-4905-86b9-121fdbb67623-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-t7blt\" (UID: \"b195df79-0181-4905-86b9-121fdbb67623\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:45.727681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.727542 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b195df79-0181-4905-86b9-121fdbb67623-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-t7blt\" (UID: \"b195df79-0181-4905-86b9-121fdbb67623\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:45.727681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.727566 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg24d\" (UniqueName: \"kubernetes.io/projected/b195df79-0181-4905-86b9-121fdbb67623-kube-api-access-mg24d\") pod \"prometheus-operator-6f4946db56-t7blt\" (UID: \"b195df79-0181-4905-86b9-121fdbb67623\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:45.727681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.727586 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b195df79-0181-4905-86b9-121fdbb67623-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-t7blt\" (UID: \"b195df79-0181-4905-86b9-121fdbb67623\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:45.727805 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:45.727701 2578 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 08 14:17:45.727852 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:45.727839 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b195df79-0181-4905-86b9-121fdbb67623-prometheus-operator-tls podName:b195df79-0181-4905-86b9-121fdbb67623 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:46.227816564 +0000 UTC m=+78.512133679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b195df79-0181-4905-86b9-121fdbb67623-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-t7blt" (UID: "b195df79-0181-4905-86b9-121fdbb67623") : secret "prometheus-operator-tls" not found Sep 08 14:17:45.728146 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.728129 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b195df79-0181-4905-86b9-121fdbb67623-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-t7blt\" (UID: \"b195df79-0181-4905-86b9-121fdbb67623\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:45.730047 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.730029 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b195df79-0181-4905-86b9-121fdbb67623-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-t7blt\" (UID: \"b195df79-0181-4905-86b9-121fdbb67623\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:45.737673 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:45.737639 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg24d\" (UniqueName: \"kubernetes.io/projected/b195df79-0181-4905-86b9-121fdbb67623-kube-api-access-mg24d\") pod \"prometheus-operator-6f4946db56-t7blt\" (UID: \"b195df79-0181-4905-86b9-121fdbb67623\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:46.230405 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:46.230362 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b195df79-0181-4905-86b9-121fdbb67623-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-t7blt\" (UID: \"b195df79-0181-4905-86b9-121fdbb67623\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:46.232953 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:46.232934 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b195df79-0181-4905-86b9-121fdbb67623-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-t7blt\" (UID: \"b195df79-0181-4905-86b9-121fdbb67623\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:46.466515 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:46.466479 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" Sep 08 14:17:46.735462 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:46.735426 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-t7blt"] Sep 08 14:17:46.738406 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:46.738372 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb195df79_0181_4905_86b9_121fdbb67623.slice/crio-6c81c8e87519cebbb926796a3ecc243be2c4821a329c0099e75ea9a36b09abf9 WatchSource:0}: Error finding container 6c81c8e87519cebbb926796a3ecc243be2c4821a329c0099e75ea9a36b09abf9: Status 404 returned error can't find the container with id 6c81c8e87519cebbb926796a3ecc243be2c4821a329c0099e75ea9a36b09abf9 Sep 08 14:17:47.750529 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:47.750485 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" event={"ID":"b195df79-0181-4905-86b9-121fdbb67623","Type":"ContainerStarted","Data":"6c81c8e87519cebbb926796a3ecc243be2c4821a329c0099e75ea9a36b09abf9"} Sep 08 14:17:48.756654 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:48.756536 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" event={"ID":"b195df79-0181-4905-86b9-121fdbb67623","Type":"ContainerStarted","Data":"f1ce1c05a1088a7435d3e8606c2a13fd725843cb2efc0456aa6a63f4037b5d4b"} Sep 08 14:17:48.756654 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:48.756589 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" event={"ID":"b195df79-0181-4905-86b9-121fdbb67623","Type":"ContainerStarted","Data":"29c611fd821b1a33f386531da4b122fec5b92c655106215947bea1f5a2245d5b"} Sep 08 14:17:48.774489 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:48.774448 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-t7blt" podStartSLOduration=2.536536198 podStartE2EDuration="3.774434977s" totalImagesPullingTime="1.237898779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:48.774403664 +0000 UTC m=+81.058720792" watchObservedRunningTime="2026-09-08 14:17:48.774434977 +0000 UTC m=+81.058752091" Sep 08 14:17:49.138848 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.138762 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d9c4545c5-6zzwc"] Sep 08 14:17:49.144615 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.144595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.147055 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.147030 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 08 14:17:49.147055 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.147052 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 08 14:17:49.147581 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.147560 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 08 14:17:49.147581 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.147579 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kd8vx\"" Sep 08 14:17:49.147778 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.147623 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 08 14:17:49.147847 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.147762 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 08 14:17:49.147847 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.147791 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 08 14:17:49.147847 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.147800 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 08 14:17:49.153117 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.153099 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 08 14:17:49.157403 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.157383 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d9c4545c5-6zzwc"] Sep 08 14:17:49.251694 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.251663 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-trusted-ca-bundle\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.251848 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.251700 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npqkd\" (UniqueName: \"kubernetes.io/projected/082744e7-08db-4d30-a96f-1d198b8cde90-kube-api-access-npqkd\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.251848 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.251749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/082744e7-08db-4d30-a96f-1d198b8cde90-console-serving-cert\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.251848 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.251788 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/082744e7-08db-4d30-a96f-1d198b8cde90-console-oauth-config\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.251848 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.251824 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-service-ca\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.251848 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.251847 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-console-config\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.252011 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.251861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-oauth-serving-cert\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.353136 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.353101 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-trusted-ca-bundle\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.353136 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.353141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npqkd\" (UniqueName: \"kubernetes.io/projected/082744e7-08db-4d30-a96f-1d198b8cde90-kube-api-access-npqkd\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.353323 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.353157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/082744e7-08db-4d30-a96f-1d198b8cde90-console-serving-cert\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.353323 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.353174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/082744e7-08db-4d30-a96f-1d198b8cde90-console-oauth-config\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.353323 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.353310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-service-ca\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.353463 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.353348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-console-config\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.353463 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.353376 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-oauth-serving-cert\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.354085 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.354062 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-service-ca\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.354183 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.354085 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-trusted-ca-bundle\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.354273 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.354253 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-oauth-serving-cert\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.354362 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.354339 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-console-config\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.355708 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.355686 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/082744e7-08db-4d30-a96f-1d198b8cde90-console-oauth-config\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.355851 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.355835 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/082744e7-08db-4d30-a96f-1d198b8cde90-console-serving-cert\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.361791 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.361770 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npqkd\" (UniqueName: \"kubernetes.io/projected/082744e7-08db-4d30-a96f-1d198b8cde90-kube-api-access-npqkd\") pod \"console-6d9c4545c5-6zzwc\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.454341 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.454276 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:49.713404 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.713378 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4wnjb" Sep 08 14:17:49.728008 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.727984 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d9c4545c5-6zzwc"] Sep 08 14:17:49.731411 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:49.731386 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082744e7_08db_4d30_a96f_1d198b8cde90.slice/crio-3e56041d5624d9ebf7177529a4369d9aa215440522fdf5502c1154dd1182c990 WatchSource:0}: Error finding container 3e56041d5624d9ebf7177529a4369d9aa215440522fdf5502c1154dd1182c990: Status 404 returned error can't find the container with id 3e56041d5624d9ebf7177529a4369d9aa215440522fdf5502c1154dd1182c990 Sep 08 14:17:49.761970 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:49.761930 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9c4545c5-6zzwc" event={"ID":"082744e7-08db-4d30-a96f-1d198b8cde90","Type":"ContainerStarted","Data":"3e56041d5624d9ebf7177529a4369d9aa215440522fdf5502c1154dd1182c990"} Sep 08 14:17:50.940909 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:50.940839 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b"] Sep 08 14:17:50.944774 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:50.944723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:50.947258 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:50.947224 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 08 14:17:50.947380 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:50.947277 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-sws24\"" Sep 08 14:17:50.947668 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:50.947646 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 08 14:17:50.955154 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:50.955133 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ptz62"] Sep 08 14:17:50.959196 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:50.959155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:50.959452 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:50.959365 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b"] Sep 08 14:17:50.961527 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:50.961501 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 08 14:17:50.961813 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:50.961633 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 08 14:17:50.961931 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:50.961913 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-w6g7w\"" Sep 08 14:17:50.962250 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:50.962137 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 08 14:17:51.071552 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.071522 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6b8a77b-7ec3-4c48-ae51-339d489290ed-sys\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.071716 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.071558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-textfile\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.071716 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.071588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/00557bb1-c1cd-469c-a493-ed035cf2b98b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-xds8b\" (UID: \"00557bb1-c1cd-469c-a493-ed035cf2b98b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:51.071716 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.071676 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-tls\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.071716 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.071708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j59f\" (UniqueName: \"kubernetes.io/projected/e6b8a77b-7ec3-4c48-ae51-339d489290ed-kube-api-access-5j59f\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.071950 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.071754 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.071950 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.071807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw9cx\" (UniqueName: \"kubernetes.io/projected/00557bb1-c1cd-469c-a493-ed035cf2b98b-kube-api-access-kw9cx\") pod \"openshift-state-metrics-657fd6c54-xds8b\" (UID: \"00557bb1-c1cd-469c-a493-ed035cf2b98b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:51.071950 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.071866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00557bb1-c1cd-469c-a493-ed035cf2b98b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-xds8b\" (UID: \"00557bb1-c1cd-469c-a493-ed035cf2b98b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:51.071950 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.071893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00557bb1-c1cd-469c-a493-ed035cf2b98b-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-xds8b\" (UID: \"00557bb1-c1cd-469c-a493-ed035cf2b98b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:51.071950 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.071919 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6b8a77b-7ec3-4c48-ae51-339d489290ed-root\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.071950 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.071945 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-wtmp\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.072161 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.071968 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b8a77b-7ec3-4c48-ae51-339d489290ed-metrics-client-ca\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.072161 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.072002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.173056 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw9cx\" (UniqueName: \"kubernetes.io/projected/00557bb1-c1cd-469c-a493-ed035cf2b98b-kube-api-access-kw9cx\") pod \"openshift-state-metrics-657fd6c54-xds8b\" (UID: \"00557bb1-c1cd-469c-a493-ed035cf2b98b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:51.173056 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00557bb1-c1cd-469c-a493-ed035cf2b98b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-xds8b\" (UID: \"00557bb1-c1cd-469c-a493-ed035cf2b98b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:51.173296 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00557bb1-c1cd-469c-a493-ed035cf2b98b-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-xds8b\" (UID: \"00557bb1-c1cd-469c-a493-ed035cf2b98b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:51.173296 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173178 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6b8a77b-7ec3-4c48-ae51-339d489290ed-root\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.173296 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173205 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-wtmp\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.173296 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173228 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b8a77b-7ec3-4c48-ae51-339d489290ed-metrics-client-ca\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.173296 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.173534 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173301 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e6b8a77b-7ec3-4c48-ae51-339d489290ed-root\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.173534 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6b8a77b-7ec3-4c48-ae51-339d489290ed-sys\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.173534 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173355 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-textfile\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.173534 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/00557bb1-c1cd-469c-a493-ed035cf2b98b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-xds8b\" (UID: \"00557bb1-c1cd-469c-a493-ed035cf2b98b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:51.173534 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-tls\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.173534 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j59f\" (UniqueName: \"kubernetes.io/projected/e6b8a77b-7ec3-4c48-ae51-339d489290ed-kube-api-access-5j59f\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.173534 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.173889 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173759 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-textfile\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.173889 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173826 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e6b8a77b-7ec3-4c48-ae51-339d489290ed-sys\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.173990 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.173930 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6b8a77b-7ec3-4c48-ae51-339d489290ed-metrics-client-ca\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.174116 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.174046 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/00557bb1-c1cd-469c-a493-ed035cf2b98b-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-xds8b\" (UID: \"00557bb1-c1cd-469c-a493-ed035cf2b98b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:51.174116 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.174056 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-wtmp\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.174116 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.174089 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.174291 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:51.174137 2578 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 08 14:17:51.174291 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:51.174195 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00557bb1-c1cd-469c-a493-ed035cf2b98b-openshift-state-metrics-tls podName:00557bb1-c1cd-469c-a493-ed035cf2b98b nodeName:}" failed. No retries permitted until 2026-09-08 14:17:51.67418231 +0000 UTC m=+83.958499404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/00557bb1-c1cd-469c-a493-ed035cf2b98b-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-xds8b" (UID: "00557bb1-c1cd-469c-a493-ed035cf2b98b") : secret "openshift-state-metrics-tls" not found Sep 08 14:17:51.176095 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.176071 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/00557bb1-c1cd-469c-a493-ed035cf2b98b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-xds8b\" (UID: \"00557bb1-c1cd-469c-a493-ed035cf2b98b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:51.176814 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.176790 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-tls\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.177208 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.177185 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e6b8a77b-7ec3-4c48-ae51-339d489290ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.182936 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.182891 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j59f\" (UniqueName: \"kubernetes.io/projected/e6b8a77b-7ec3-4c48-ae51-339d489290ed-kube-api-access-5j59f\") pod \"node-exporter-ptz62\" (UID: \"e6b8a77b-7ec3-4c48-ae51-339d489290ed\") " pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.183081 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.183063 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw9cx\" (UniqueName: \"kubernetes.io/projected/00557bb1-c1cd-469c-a493-ed035cf2b98b-kube-api-access-kw9cx\") pod \"openshift-state-metrics-657fd6c54-xds8b\" (UID: \"00557bb1-c1cd-469c-a493-ed035cf2b98b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:51.270907 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.270871 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ptz62" Sep 08 14:17:51.677238 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.677149 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/00557bb1-c1cd-469c-a493-ed035cf2b98b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-xds8b\" (UID: \"00557bb1-c1cd-469c-a493-ed035cf2b98b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:51.680293 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.680265 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/00557bb1-c1cd-469c-a493-ed035cf2b98b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-xds8b\" (UID: \"00557bb1-c1cd-469c-a493-ed035cf2b98b\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:51.856680 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:51.856466 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" Sep 08 14:17:52.371159 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:52.371129 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b8a77b_7ec3_4c48_ae51_339d489290ed.slice/crio-82aedd48146cbd6f2cb958116c2c747b49bc31c8b42ce5b84c89aff3ef775f3c WatchSource:0}: Error finding container 82aedd48146cbd6f2cb958116c2c747b49bc31c8b42ce5b84c89aff3ef775f3c: Status 404 returned error can't find the container with id 82aedd48146cbd6f2cb958116c2c747b49bc31c8b42ce5b84c89aff3ef775f3c Sep 08 14:17:52.646589 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:52.646509 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b"] Sep 08 14:17:52.651093 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:52.651064 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00557bb1_c1cd_469c_a493_ed035cf2b98b.slice/crio-8b4fa6e2d7572394a66f368afe9e817df2897ca8c8da9949e17d00bcb7016bed WatchSource:0}: Error finding container 8b4fa6e2d7572394a66f368afe9e817df2897ca8c8da9949e17d00bcb7016bed: Status 404 returned error can't find the container with id 8b4fa6e2d7572394a66f368afe9e817df2897ca8c8da9949e17d00bcb7016bed Sep 08 14:17:52.779141 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:52.779111 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9c4545c5-6zzwc" event={"ID":"082744e7-08db-4d30-a96f-1d198b8cde90","Type":"ContainerStarted","Data":"7981730b129e566fff411fcd0ee24f7403d3a87facef698d3f5fba7dc425b902"} Sep 08 14:17:52.781262 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:52.781240 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" event={"ID":"00557bb1-c1cd-469c-a493-ed035cf2b98b","Type":"ContainerStarted","Data":"1250ab5f0807311dd7fc0bc08b356ead9bcc0b16bdebd067f58df40d4e43ee27"} Sep 08 14:17:52.781339 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:52.781298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" event={"ID":"00557bb1-c1cd-469c-a493-ed035cf2b98b","Type":"ContainerStarted","Data":"8b4fa6e2d7572394a66f368afe9e817df2897ca8c8da9949e17d00bcb7016bed"} Sep 08 14:17:52.782923 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:52.782896 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptz62" event={"ID":"e6b8a77b-7ec3-4c48-ae51-339d489290ed","Type":"ContainerStarted","Data":"82aedd48146cbd6f2cb958116c2c747b49bc31c8b42ce5b84c89aff3ef775f3c"} Sep 08 14:17:52.799830 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:52.799773 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d9c4545c5-6zzwc" podStartSLOduration=1.112759723 podStartE2EDuration="3.799752021s" totalImagesPullingTime="2.686992298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:52.7992422 +0000 UTC m=+85.083559316" watchObservedRunningTime="2026-09-08 14:17:52.799752021 +0000 UTC m=+85.084069137" Sep 08 14:17:53.790635 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:53.790596 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" event={"ID":"00557bb1-c1cd-469c-a493-ed035cf2b98b","Type":"ContainerStarted","Data":"3fb2a75bb2ad1b09af9459e70c919c5dee200cfed4421102d9e7973f1f7621e9"} Sep 08 14:17:54.796690 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:54.796654 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" event={"ID":"00557bb1-c1cd-469c-a493-ed035cf2b98b","Type":"ContainerStarted","Data":"8ae08230024f160cee6771c46eb1b8e5a9f4aed0fe77f50987f4817ea0b14c8a"} Sep 08 14:17:54.798384 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:54.798361 2578 generic.go:356] "Generic (PLEG): container finished" podID="e6b8a77b-7ec3-4c48-ae51-339d489290ed" containerID="9af873d955cb03791911246a84db3178fe9c4b7b3ec2dcf49c683298346b7f6b" exitCode=0 Sep 08 14:17:54.798486 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:54.798397 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptz62" event={"ID":"e6b8a77b-7ec3-4c48-ae51-339d489290ed","Type":"ContainerDied","Data":"9af873d955cb03791911246a84db3178fe9c4b7b3ec2dcf49c683298346b7f6b"} Sep 08 14:17:54.817303 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:54.817253 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-xds8b" podStartSLOduration=3.486678165 podStartE2EDuration="4.817234858s" totalImagesPullingTime="1.330556693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:54.81576206 +0000 UTC m=+87.100079172" watchObservedRunningTime="2026-09-08 14:17:54.817234858 +0000 UTC m=+87.101551973" Sep 08 14:17:55.276389 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.276342 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-689794b998-v54kh"] Sep 08 14:17:55.279541 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.279519 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.281949 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.281932 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 08 14:17:55.282112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.281934 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 08 14:17:55.282222 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.282208 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 08 14:17:55.282287 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.282276 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 08 14:17:55.282592 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.282571 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-f0qk5v66tcdlt\"" Sep 08 14:17:55.282699 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.282579 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6mlbz\"" Sep 08 14:17:55.290363 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.290341 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-689794b998-v54kh"] Sep 08 14:17:55.409903 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.409871 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/269e0a66-cd9b-442e-a6f3-7b83bca179cb-secret-metrics-server-client-certs\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.409903 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.409908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/269e0a66-cd9b-442e-a6f3-7b83bca179cb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.410149 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.409946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/269e0a66-cd9b-442e-a6f3-7b83bca179cb-metrics-server-audit-profiles\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.410149 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.410033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/269e0a66-cd9b-442e-a6f3-7b83bca179cb-audit-log\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.410149 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.410058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6764l\" (UniqueName: \"kubernetes.io/projected/269e0a66-cd9b-442e-a6f3-7b83bca179cb-kube-api-access-6764l\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.410149 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.410094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/269e0a66-cd9b-442e-a6f3-7b83bca179cb-secret-metrics-server-tls\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.410318 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.410177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/269e0a66-cd9b-442e-a6f3-7b83bca179cb-client-ca-bundle\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.510866 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.510828 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/269e0a66-cd9b-442e-a6f3-7b83bca179cb-audit-log\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.510866 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.510863 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6764l\" (UniqueName: \"kubernetes.io/projected/269e0a66-cd9b-442e-a6f3-7b83bca179cb-kube-api-access-6764l\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.511122 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.510886 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/269e0a66-cd9b-442e-a6f3-7b83bca179cb-secret-metrics-server-tls\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.511122 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.510910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/269e0a66-cd9b-442e-a6f3-7b83bca179cb-client-ca-bundle\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.511122 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.510937 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/269e0a66-cd9b-442e-a6f3-7b83bca179cb-secret-metrics-server-client-certs\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.511122 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.510962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/269e0a66-cd9b-442e-a6f3-7b83bca179cb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.511122 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.511022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/269e0a66-cd9b-442e-a6f3-7b83bca179cb-metrics-server-audit-profiles\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.511356 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.511340 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/269e0a66-cd9b-442e-a6f3-7b83bca179cb-audit-log\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.511899 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.511832 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/269e0a66-cd9b-442e-a6f3-7b83bca179cb-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.512136 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.512116 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/269e0a66-cd9b-442e-a6f3-7b83bca179cb-metrics-server-audit-profiles\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.514115 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.514089 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/269e0a66-cd9b-442e-a6f3-7b83bca179cb-secret-metrics-server-client-certs\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.514215 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.514190 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/269e0a66-cd9b-442e-a6f3-7b83bca179cb-client-ca-bundle\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.514283 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.514244 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/269e0a66-cd9b-442e-a6f3-7b83bca179cb-secret-metrics-server-tls\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.520207 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.520186 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6764l\" (UniqueName: \"kubernetes.io/projected/269e0a66-cd9b-442e-a6f3-7b83bca179cb-kube-api-access-6764l\") pod \"metrics-server-689794b998-v54kh\" (UID: \"269e0a66-cd9b-442e-a6f3-7b83bca179cb\") " pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.588572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.588495 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:17:55.686920 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.686892 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77bc4bb45d-c2pxl" Sep 08 14:17:55.724454 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.724412 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk"] Sep 08 14:17:55.729237 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.729215 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" Sep 08 14:17:55.731412 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.731387 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 08 14:17:55.731523 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.731448 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gvshq\"" Sep 08 14:17:55.735031 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.735007 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk"] Sep 08 14:17:55.804040 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.804003 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptz62" event={"ID":"e6b8a77b-7ec3-4c48-ae51-339d489290ed","Type":"ContainerStarted","Data":"8aa23639b2a2631b3fe3587a17e17083ae28ed62fe8906da9bc497f8a72a028c"} Sep 08 14:17:55.804400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.804055 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ptz62" event={"ID":"e6b8a77b-7ec3-4c48-ae51-339d489290ed","Type":"ContainerStarted","Data":"f0f136919d96eb19e6186350c6f904c314f4f3ad57f90b9cc434a062defa51e7"} Sep 08 14:17:55.812886 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.812867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qfrh\" (UniqueName: \"kubernetes.io/projected/6cf0f337-7106-4c18-9e39-2725ca0e8572-kube-api-access-8qfrh\") pod \"monitoring-plugin-85bbbbbf4-t56nk\" (UID: \"6cf0f337-7106-4c18-9e39-2725ca0e8572\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" Sep 08 14:17:55.812994 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.812972 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6cf0f337-7106-4c18-9e39-2725ca0e8572-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t56nk\" (UID: \"6cf0f337-7106-4c18-9e39-2725ca0e8572\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" Sep 08 14:17:55.822939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.822897 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ptz62" podStartSLOduration=3.515332183 podStartE2EDuration="5.822883276s" totalImagesPullingTime="2.237166219s" totalInitContainerRuntime="70.384874ms" isStatefulPod=true podCreationTimestamp="2026-09-08 14:17:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:55.821484463 +0000 UTC m=+88.105801578" watchObservedRunningTime="2026-09-08 14:17:55.822883276 +0000 UTC m=+88.107200392" Sep 08 14:17:55.870667 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.870640 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-689794b998-v54kh"] Sep 08 14:17:55.873139 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:55.873116 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod269e0a66_cd9b_442e_a6f3_7b83bca179cb.slice/crio-85dc60b5ceb1158ffe0075557d0752d344b47971f2188cd01bd5717f55b0cfd7 WatchSource:0}: Error finding container 85dc60b5ceb1158ffe0075557d0752d344b47971f2188cd01bd5717f55b0cfd7: Status 404 returned error can't find the container with id 85dc60b5ceb1158ffe0075557d0752d344b47971f2188cd01bd5717f55b0cfd7 Sep 08 14:17:55.914293 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.914265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qfrh\" (UniqueName: \"kubernetes.io/projected/6cf0f337-7106-4c18-9e39-2725ca0e8572-kube-api-access-8qfrh\") pod \"monitoring-plugin-85bbbbbf4-t56nk\" (UID: \"6cf0f337-7106-4c18-9e39-2725ca0e8572\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" Sep 08 14:17:55.914485 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.914459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6cf0f337-7106-4c18-9e39-2725ca0e8572-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t56nk\" (UID: \"6cf0f337-7106-4c18-9e39-2725ca0e8572\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" Sep 08 14:17:55.914703 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:55.914684 2578 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 08 14:17:55.914815 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:17:55.914783 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cf0f337-7106-4c18-9e39-2725ca0e8572-monitoring-plugin-cert podName:6cf0f337-7106-4c18-9e39-2725ca0e8572 nodeName:}" failed. No retries permitted until 2026-09-08 14:17:56.414761351 +0000 UTC m=+88.699078444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/6cf0f337-7106-4c18-9e39-2725ca0e8572-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-t56nk" (UID: "6cf0f337-7106-4c18-9e39-2725ca0e8572") : secret "monitoring-plugin-cert" not found Sep 08 14:17:55.925630 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:55.925609 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qfrh\" (UniqueName: \"kubernetes.io/projected/6cf0f337-7106-4c18-9e39-2725ca0e8572-kube-api-access-8qfrh\") pod \"monitoring-plugin-85bbbbbf4-t56nk\" (UID: \"6cf0f337-7106-4c18-9e39-2725ca0e8572\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" Sep 08 14:17:56.181262 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.181176 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-98d849dd-mzr8j"] Sep 08 14:17:56.186760 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.185374 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.188814 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.188787 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6vb8d\"" Sep 08 14:17:56.188814 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.188808 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 08 14:17:56.188993 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.188799 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 08 14:17:56.188993 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.188787 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 08 14:17:56.189068 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.189044 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 08 14:17:56.189178 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.189163 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 08 14:17:56.197512 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.197492 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 08 14:17:56.199808 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.199780 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-98d849dd-mzr8j"] Sep 08 14:17:56.318561 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.318530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea434857-4c11-47cc-8b4e-ac465691ba3a-metrics-client-ca\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.318561 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.318563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea434857-4c11-47cc-8b4e-ac465691ba3a-secret-telemeter-client\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.318781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.318586 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea434857-4c11-47cc-8b4e-ac465691ba3a-federate-client-tls\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.318781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.318601 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea434857-4c11-47cc-8b4e-ac465691ba3a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.318781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.318660 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea434857-4c11-47cc-8b4e-ac465691ba3a-serving-certs-ca-bundle\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.318781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.318758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea434857-4c11-47cc-8b4e-ac465691ba3a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.318911 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.318802 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjbdj\" (UniqueName: \"kubernetes.io/projected/ea434857-4c11-47cc-8b4e-ac465691ba3a-kube-api-access-fjbdj\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.318911 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.318838 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea434857-4c11-47cc-8b4e-ac465691ba3a-telemeter-client-tls\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.420070 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.420035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjbdj\" (UniqueName: \"kubernetes.io/projected/ea434857-4c11-47cc-8b4e-ac465691ba3a-kube-api-access-fjbdj\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.420070 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.420078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6cf0f337-7106-4c18-9e39-2725ca0e8572-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t56nk\" (UID: \"6cf0f337-7106-4c18-9e39-2725ca0e8572\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" Sep 08 14:17:56.420364 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.420097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea434857-4c11-47cc-8b4e-ac465691ba3a-telemeter-client-tls\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.420364 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.420119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea434857-4c11-47cc-8b4e-ac465691ba3a-metrics-client-ca\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.420364 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.420138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea434857-4c11-47cc-8b4e-ac465691ba3a-secret-telemeter-client\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.420364 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.420165 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea434857-4c11-47cc-8b4e-ac465691ba3a-federate-client-tls\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.420364 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.420293 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea434857-4c11-47cc-8b4e-ac465691ba3a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.420364 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.420348 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea434857-4c11-47cc-8b4e-ac465691ba3a-serving-certs-ca-bundle\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.420656 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.420424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea434857-4c11-47cc-8b4e-ac465691ba3a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.421260 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.421235 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ea434857-4c11-47cc-8b4e-ac465691ba3a-metrics-client-ca\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.421376 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.421334 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea434857-4c11-47cc-8b4e-ac465691ba3a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.421376 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.421367 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ea434857-4c11-47cc-8b4e-ac465691ba3a-serving-certs-ca-bundle\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.423099 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.423075 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ea434857-4c11-47cc-8b4e-ac465691ba3a-telemeter-client-tls\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.423536 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.423516 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ea434857-4c11-47cc-8b4e-ac465691ba3a-secret-telemeter-client\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.423586 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.423516 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6cf0f337-7106-4c18-9e39-2725ca0e8572-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t56nk\" (UID: \"6cf0f337-7106-4c18-9e39-2725ca0e8572\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" Sep 08 14:17:56.423678 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.423656 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ea434857-4c11-47cc-8b4e-ac465691ba3a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.423710 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.423688 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ea434857-4c11-47cc-8b4e-ac465691ba3a-federate-client-tls\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.428647 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.428623 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjbdj\" (UniqueName: \"kubernetes.io/projected/ea434857-4c11-47cc-8b4e-ac465691ba3a-kube-api-access-fjbdj\") pod \"telemeter-client-98d849dd-mzr8j\" (UID: \"ea434857-4c11-47cc-8b4e-ac465691ba3a\") " pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.496668 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.496629 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" Sep 08 14:17:56.640590 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.640560 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" Sep 08 14:17:56.772540 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.772517 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-98d849dd-mzr8j"] Sep 08 14:17:56.774603 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:56.774579 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea434857_4c11_47cc_8b4e_ac465691ba3a.slice/crio-1f65a2a25b1d52f2a2a47ad73edddc16e5a188c0a36f9b9e8bcba3ca2c4e66bf WatchSource:0}: Error finding container 1f65a2a25b1d52f2a2a47ad73edddc16e5a188c0a36f9b9e8bcba3ca2c4e66bf: Status 404 returned error can't find the container with id 1f65a2a25b1d52f2a2a47ad73edddc16e5a188c0a36f9b9e8bcba3ca2c4e66bf Sep 08 14:17:56.809408 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.809370 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" event={"ID":"ea434857-4c11-47cc-8b4e-ac465691ba3a","Type":"ContainerStarted","Data":"1f65a2a25b1d52f2a2a47ad73edddc16e5a188c0a36f9b9e8bcba3ca2c4e66bf"} Sep 08 14:17:56.810555 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.810534 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-689794b998-v54kh" event={"ID":"269e0a66-cd9b-442e-a6f3-7b83bca179cb","Type":"ContainerStarted","Data":"85dc60b5ceb1158ffe0075557d0752d344b47971f2188cd01bd5717f55b0cfd7"} Sep 08 14:17:56.913076 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:56.912998 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk"] Sep 08 14:17:56.917036 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:17:56.917011 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf0f337_7106_4c18_9e39_2725ca0e8572.slice/crio-b7bf741c46b133df960b14a3d877abbde221aded38b9f1a057716deb9b1beb6d WatchSource:0}: Error finding container b7bf741c46b133df960b14a3d877abbde221aded38b9f1a057716deb9b1beb6d: Status 404 returned error can't find the container with id b7bf741c46b133df960b14a3d877abbde221aded38b9f1a057716deb9b1beb6d Sep 08 14:17:57.816377 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:57.816338 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" event={"ID":"6cf0f337-7106-4c18-9e39-2725ca0e8572","Type":"ContainerStarted","Data":"b7bf741c46b133df960b14a3d877abbde221aded38b9f1a057716deb9b1beb6d"} Sep 08 14:17:59.454844 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:59.454812 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:59.455233 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:59.455041 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:59.459886 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:59.459867 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:59.824891 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:59.824848 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" event={"ID":"6cf0f337-7106-4c18-9e39-2725ca0e8572","Type":"ContainerStarted","Data":"3d4ebcd0b142d70316eecd4bda7233ec960721cf6325d81fbb3be32fe089dd96"} Sep 08 14:17:59.825173 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:59.824913 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" Sep 08 14:17:59.826383 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:59.826360 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" event={"ID":"ea434857-4c11-47cc-8b4e-ac465691ba3a","Type":"ContainerStarted","Data":"dfc185702d35a51ae2de346c574634da1a5d64e9184fe72a41418da73c98606d"} Sep 08 14:17:59.831178 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:59.831159 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" Sep 08 14:17:59.831584 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:59.831565 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:17:59.839252 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:17:59.839216 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t56nk" podStartSLOduration=2.96285254 podStartE2EDuration="4.839202387s" totalImagesPullingTime="1.876349847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:17:59.838845391 +0000 UTC m=+92.123162506" watchObservedRunningTime="2026-09-08 14:17:59.839202387 +0000 UTC m=+92.123519546" Sep 08 14:18:00.129579 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.129500 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59f5dc65b8-8qvjg"] Sep 08 14:18:00.133194 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.133169 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.144793 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.144708 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59f5dc65b8-8qvjg"] Sep 08 14:18:00.256637 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.256295 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2149aba-5979-43c9-aba4-6c698d36eb86-console-serving-cert\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.256637 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.256330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2149aba-5979-43c9-aba4-6c698d36eb86-console-oauth-config\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.256637 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.256356 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcw4x\" (UniqueName: \"kubernetes.io/projected/e2149aba-5979-43c9-aba4-6c698d36eb86-kube-api-access-vcw4x\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.256637 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.256376 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-console-config\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.256637 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.256452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-trusted-ca-bundle\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.256637 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.256501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-service-ca\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.256637 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.256548 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-oauth-serving-cert\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.358174 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.357796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-oauth-serving-cert\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.358174 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.357872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2149aba-5979-43c9-aba4-6c698d36eb86-console-serving-cert\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.358174 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.357900 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2149aba-5979-43c9-aba4-6c698d36eb86-console-oauth-config\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.358174 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.357928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcw4x\" (UniqueName: \"kubernetes.io/projected/e2149aba-5979-43c9-aba4-6c698d36eb86-kube-api-access-vcw4x\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.358174 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.357958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-console-config\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.358174 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.358005 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-trusted-ca-bundle\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.358174 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.358052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-service-ca\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.358658 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.358603 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-oauth-serving-cert\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.358998 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.358970 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-console-config\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.359474 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.359449 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-trusted-ca-bundle\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.359588 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.359527 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-service-ca\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.361258 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.361232 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2149aba-5979-43c9-aba4-6c698d36eb86-console-serving-cert\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.361819 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.361801 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2149aba-5979-43c9-aba4-6c698d36eb86-console-oauth-config\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.368707 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.368678 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcw4x\" (UniqueName: \"kubernetes.io/projected/e2149aba-5979-43c9-aba4-6c698d36eb86-kube-api-access-vcw4x\") pod \"console-59f5dc65b8-8qvjg\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:00.449976 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:00.449885 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:01.371755 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:01.371656 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59f5dc65b8-8qvjg"] Sep 08 14:18:01.375990 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:18:01.375964 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2149aba_5979_43c9_aba4_6c698d36eb86.slice/crio-959e2dc08ae3bee383ccdc6fd03896a5c077799c7a21c843f4d10df41176643b WatchSource:0}: Error finding container 959e2dc08ae3bee383ccdc6fd03896a5c077799c7a21c843f4d10df41176643b: Status 404 returned error can't find the container with id 959e2dc08ae3bee383ccdc6fd03896a5c077799c7a21c843f4d10df41176643b Sep 08 14:18:01.837590 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:01.837550 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f5dc65b8-8qvjg" event={"ID":"e2149aba-5979-43c9-aba4-6c698d36eb86","Type":"ContainerStarted","Data":"73a50479b2b516c39d4a32a98a149eb7c750b623d135e821ab949c8813d03d48"} Sep 08 14:18:01.837794 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:01.837610 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f5dc65b8-8qvjg" event={"ID":"e2149aba-5979-43c9-aba4-6c698d36eb86","Type":"ContainerStarted","Data":"959e2dc08ae3bee383ccdc6fd03896a5c077799c7a21c843f4d10df41176643b"} Sep 08 14:18:01.839786 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:01.839760 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" event={"ID":"ea434857-4c11-47cc-8b4e-ac465691ba3a","Type":"ContainerStarted","Data":"d22974f51afff75972ca9a5d0df187a294f2cff7242a8a51f24e38ad25559238"} Sep 08 14:18:01.839908 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:01.839822 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" event={"ID":"ea434857-4c11-47cc-8b4e-ac465691ba3a","Type":"ContainerStarted","Data":"1d8b3eeeb12f108a50da4c7df621f97d3d5b35b434ffdd612dd7e2a5c3fd4934"} Sep 08 14:18:01.841368 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:01.841325 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-689794b998-v54kh" event={"ID":"269e0a66-cd9b-442e-a6f3-7b83bca179cb","Type":"ContainerStarted","Data":"8f96b243453861736fbf1da941997a003462fabdefa4b848cbafaea2b6b058b6"} Sep 08 14:18:01.855675 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:01.855633 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59f5dc65b8-8qvjg" podStartSLOduration=1.8556165789999999 podStartE2EDuration="1.855616579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:18:01.854778114 +0000 UTC m=+94.139095229" watchObservedRunningTime="2026-09-08 14:18:01.855616579 +0000 UTC m=+94.139933696" Sep 08 14:18:01.871942 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:01.871890 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-689794b998-v54kh" podStartSLOduration=1.6536433069999998 podStartE2EDuration="6.871870183s" totalImagesPullingTime="5.218226876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:18:01.871487721 +0000 UTC m=+94.155804844" watchObservedRunningTime="2026-09-08 14:18:01.871870183 +0000 UTC m=+94.156187301" Sep 08 14:18:01.894016 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:01.893959 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-98d849dd-mzr8j" podStartSLOduration=1.638206599 podStartE2EDuration="5.893935813s" totalImagesPullingTime="4.255729214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:17:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:18:01.892096424 +0000 UTC m=+94.176413535" watchObservedRunningTime="2026-09-08 14:18:01.893935813 +0000 UTC m=+94.178252928" Sep 08 14:18:02.134752 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.134651 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59f5dc65b8-8qvjg"] Sep 08 14:18:02.165567 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.165534 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ddf646b7f-74fz8"] Sep 08 14:18:02.169560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.169537 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.180275 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.180251 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ddf646b7f-74fz8"] Sep 08 14:18:02.276471 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.276416 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-serving-cert\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.276695 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.276675 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-oauth-config\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.276847 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.276830 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-config\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.276954 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.276940 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-service-ca\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.277057 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.277045 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-trusted-ca-bundle\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.277161 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.277149 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-oauth-serving-cert\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.277275 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.277263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pszg\" (UniqueName: \"kubernetes.io/projected/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-kube-api-access-2pszg\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.377818 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.377777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pszg\" (UniqueName: \"kubernetes.io/projected/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-kube-api-access-2pszg\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.378223 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.377861 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-serving-cert\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.378223 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.377879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-oauth-config\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.378223 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.377895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-config\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.378223 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.377912 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-service-ca\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.378223 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.377930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-trusted-ca-bundle\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.378223 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.378134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-oauth-serving-cert\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.378913 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.378887 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-oauth-serving-cert\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.379050 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.378939 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-trusted-ca-bundle\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.379050 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.378971 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-config\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.379050 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.378999 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-service-ca\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.380694 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.380671 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-serving-cert\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.380822 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.380805 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-oauth-config\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.385850 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.385806 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pszg\" (UniqueName: \"kubernetes.io/projected/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-kube-api-access-2pszg\") pod \"console-ddf646b7f-74fz8\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.479255 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.479213 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:02.757614 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.757502 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ddf646b7f-74fz8"] Sep 08 14:18:02.760195 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:18:02.760165 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b880e6_3b0b_4a55_8b41_bd8189a4f1f0.slice/crio-79f82f25984435ea17a68a7c67604eac433edd057a30f229dbeb4a827fe72c74 WatchSource:0}: Error finding container 79f82f25984435ea17a68a7c67604eac433edd057a30f229dbeb4a827fe72c74: Status 404 returned error can't find the container with id 79f82f25984435ea17a68a7c67604eac433edd057a30f229dbeb4a827fe72c74 Sep 08 14:18:02.849226 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.849183 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddf646b7f-74fz8" event={"ID":"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0","Type":"ContainerStarted","Data":"8c2ae8fab951cd4d82402ed8c56f7a1ba1580b80132a3d57536aecf7f5725dd7"} Sep 08 14:18:02.849491 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.849243 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddf646b7f-74fz8" event={"ID":"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0","Type":"ContainerStarted","Data":"79f82f25984435ea17a68a7c67604eac433edd057a30f229dbeb4a827fe72c74"} Sep 08 14:18:02.867567 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:02.867512 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ddf646b7f-74fz8" podStartSLOduration=0.867495714 podStartE2EDuration="867.495714ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:18:02.866447835 +0000 UTC m=+95.150764951" watchObservedRunningTime="2026-09-08 14:18:02.867495714 +0000 UTC m=+95.151812829" Sep 08 14:18:10.451131 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:10.451092 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:10.714502 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:10.714469 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4njj8" Sep 08 14:18:12.479965 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:12.479932 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:12.480400 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:12.479987 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:12.484936 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:12.484912 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:12.895575 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:12.895490 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:18:12.945422 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:12.945394 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d9c4545c5-6zzwc"] Sep 08 14:18:15.589439 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:15.589401 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:18:15.590033 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:15.589471 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:18:28.873877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:28.873813 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59f5dc65b8-8qvjg" podUID="e2149aba-5979-43c9-aba4-6c698d36eb86" containerName="console" containerID="cri-o://73a50479b2b516c39d4a32a98a149eb7c750b623d135e821ab949c8813d03d48" gracePeriod=15 Sep 08 14:18:29.111918 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.111894 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:29.210450 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.210418 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2149aba-5979-43c9-aba4-6c698d36eb86-console-serving-cert\") pod \"e2149aba-5979-43c9-aba4-6c698d36eb86\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " Sep 08 14:18:29.210627 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.210459 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-oauth-serving-cert\") pod \"e2149aba-5979-43c9-aba4-6c698d36eb86\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " Sep 08 14:18:29.210627 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.210485 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-console-config\") pod \"e2149aba-5979-43c9-aba4-6c698d36eb86\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " Sep 08 14:18:29.210627 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.210514 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcw4x\" (UniqueName: \"kubernetes.io/projected/e2149aba-5979-43c9-aba4-6c698d36eb86-kube-api-access-vcw4x\") pod \"e2149aba-5979-43c9-aba4-6c698d36eb86\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " Sep 08 14:18:29.210627 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.210578 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-trusted-ca-bundle\") pod \"e2149aba-5979-43c9-aba4-6c698d36eb86\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " Sep 08 14:18:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.210653 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2149aba-5979-43c9-aba4-6c698d36eb86-console-oauth-config\") pod \"e2149aba-5979-43c9-aba4-6c698d36eb86\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " Sep 08 14:18:29.210877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.210680 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-service-ca\") pod \"e2149aba-5979-43c9-aba4-6c698d36eb86\" (UID: \"e2149aba-5979-43c9-aba4-6c698d36eb86\") " Sep 08 14:18:29.210973 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.210935 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e2149aba-5979-43c9-aba4-6c698d36eb86" (UID: "e2149aba-5979-43c9-aba4-6c698d36eb86"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:18:29.210973 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.210945 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-console-config" (OuterVolumeSpecName: "console-config") pod "e2149aba-5979-43c9-aba4-6c698d36eb86" (UID: "e2149aba-5979-43c9-aba4-6c698d36eb86"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:18:29.211138 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.211109 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e2149aba-5979-43c9-aba4-6c698d36eb86" (UID: "e2149aba-5979-43c9-aba4-6c698d36eb86"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:18:29.211257 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.211188 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-service-ca" (OuterVolumeSpecName: "service-ca") pod "e2149aba-5979-43c9-aba4-6c698d36eb86" (UID: "e2149aba-5979-43c9-aba4-6c698d36eb86"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:18:29.212931 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.212902 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2149aba-5979-43c9-aba4-6c698d36eb86-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e2149aba-5979-43c9-aba4-6c698d36eb86" (UID: "e2149aba-5979-43c9-aba4-6c698d36eb86"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:18:29.212931 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.212919 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2149aba-5979-43c9-aba4-6c698d36eb86-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e2149aba-5979-43c9-aba4-6c698d36eb86" (UID: "e2149aba-5979-43c9-aba4-6c698d36eb86"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:18:29.213056 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.213002 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2149aba-5979-43c9-aba4-6c698d36eb86-kube-api-access-vcw4x" (OuterVolumeSpecName: "kube-api-access-vcw4x") pod "e2149aba-5979-43c9-aba4-6c698d36eb86" (UID: "e2149aba-5979-43c9-aba4-6c698d36eb86"). InnerVolumeSpecName "kube-api-access-vcw4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:18:29.311615 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.311577 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-console-config\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:29.311615 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.311610 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcw4x\" (UniqueName: \"kubernetes.io/projected/e2149aba-5979-43c9-aba4-6c698d36eb86-kube-api-access-vcw4x\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:29.311615 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.311619 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-trusted-ca-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:29.311615 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.311627 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e2149aba-5979-43c9-aba4-6c698d36eb86-console-oauth-config\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:29.311902 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.311636 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:29.311902 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.311644 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e2149aba-5979-43c9-aba4-6c698d36eb86-console-serving-cert\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:29.311902 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.311651 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e2149aba-5979-43c9-aba4-6c698d36eb86-oauth-serving-cert\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:29.968782 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.968749 2578 generic.go:356] "Generic (PLEG): container finished" podID="e2149aba-5979-43c9-aba4-6c698d36eb86" containerID="73a50479b2b516c39d4a32a98a149eb7c750b623d135e821ab949c8813d03d48" exitCode=0 Sep 08 14:18:29.969179 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.968794 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f5dc65b8-8qvjg" Sep 08 14:18:29.969179 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.968795 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f5dc65b8-8qvjg" event={"ID":"e2149aba-5979-43c9-aba4-6c698d36eb86","Type":"ContainerDied","Data":"73a50479b2b516c39d4a32a98a149eb7c750b623d135e821ab949c8813d03d48"} Sep 08 14:18:29.969179 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.968932 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f5dc65b8-8qvjg" event={"ID":"e2149aba-5979-43c9-aba4-6c698d36eb86","Type":"ContainerDied","Data":"959e2dc08ae3bee383ccdc6fd03896a5c077799c7a21c843f4d10df41176643b"} Sep 08 14:18:29.969179 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.968961 2578 scope.go:117] "RemoveContainer" containerID="73a50479b2b516c39d4a32a98a149eb7c750b623d135e821ab949c8813d03d48" Sep 08 14:18:29.978697 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.978680 2578 scope.go:117] "RemoveContainer" containerID="73a50479b2b516c39d4a32a98a149eb7c750b623d135e821ab949c8813d03d48" Sep 08 14:18:29.979036 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:18:29.979014 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a50479b2b516c39d4a32a98a149eb7c750b623d135e821ab949c8813d03d48\": container with ID starting with 73a50479b2b516c39d4a32a98a149eb7c750b623d135e821ab949c8813d03d48 not found: ID does not exist" containerID="73a50479b2b516c39d4a32a98a149eb7c750b623d135e821ab949c8813d03d48" Sep 08 14:18:29.979109 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.979046 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a50479b2b516c39d4a32a98a149eb7c750b623d135e821ab949c8813d03d48"} err="failed to get container status \"73a50479b2b516c39d4a32a98a149eb7c750b623d135e821ab949c8813d03d48\": rpc error: code = NotFound desc = could not find container \"73a50479b2b516c39d4a32a98a149eb7c750b623d135e821ab949c8813d03d48\": container with ID starting with 73a50479b2b516c39d4a32a98a149eb7c750b623d135e821ab949c8813d03d48 not found: ID does not exist" Sep 08 14:18:29.990654 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.990625 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59f5dc65b8-8qvjg"] Sep 08 14:18:29.994830 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:29.994806 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59f5dc65b8-8qvjg"] Sep 08 14:18:30.240213 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:30.240136 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2149aba-5979-43c9-aba4-6c698d36eb86" path="/var/lib/kubelet/pods/e2149aba-5979-43c9-aba4-6c698d36eb86/volumes" Sep 08 14:18:34.992503 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:34.992465 2578 generic.go:356] "Generic (PLEG): container finished" podID="bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55" containerID="d99de94d63fd7a1a390fd30ddf41a8be84deb7346abbb59579a0a0d016c3a105" exitCode=0 Sep 08 14:18:34.992898 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:34.992534 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" event={"ID":"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55","Type":"ContainerDied","Data":"d99de94d63fd7a1a390fd30ddf41a8be84deb7346abbb59579a0a0d016c3a105"} Sep 08 14:18:34.992945 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:34.992907 2578 scope.go:117] "RemoveContainer" containerID="d99de94d63fd7a1a390fd30ddf41a8be84deb7346abbb59579a0a0d016c3a105" Sep 08 14:18:35.594507 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:35.594476 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:18:35.598388 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:35.598357 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-689794b998-v54kh" Sep 08 14:18:35.629175 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:35.629145 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pxs2b_38343840-71de-4d89-b76f-d628180edba4/serve-healthcheck-canary/0.log" Sep 08 14:18:35.999887 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:35.999852 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-fpwss" event={"ID":"bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55","Type":"ContainerStarted","Data":"117f6d4c75acd99e70581003c12f8fe7171a2531f06f66e6210249e59a6dd05a"} Sep 08 14:18:37.966182 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:37.966133 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d9c4545c5-6zzwc" podUID="082744e7-08db-4d30-a96f-1d198b8cde90" containerName="console" containerID="cri-o://7981730b129e566fff411fcd0ee24f7403d3a87facef698d3f5fba7dc425b902" gracePeriod=15 Sep 08 14:18:38.211431 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.211409 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:18:38.291916 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.291827 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-trusted-ca-bundle\") pod \"082744e7-08db-4d30-a96f-1d198b8cde90\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " Sep 08 14:18:38.291916 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.291882 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-console-config\") pod \"082744e7-08db-4d30-a96f-1d198b8cde90\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " Sep 08 14:18:38.292133 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.291929 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npqkd\" (UniqueName: \"kubernetes.io/projected/082744e7-08db-4d30-a96f-1d198b8cde90-kube-api-access-npqkd\") pod \"082744e7-08db-4d30-a96f-1d198b8cde90\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " Sep 08 14:18:38.292133 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.291950 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/082744e7-08db-4d30-a96f-1d198b8cde90-console-serving-cert\") pod \"082744e7-08db-4d30-a96f-1d198b8cde90\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " Sep 08 14:18:38.292133 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.291970 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-service-ca\") pod \"082744e7-08db-4d30-a96f-1d198b8cde90\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " Sep 08 14:18:38.292133 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.291993 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-oauth-serving-cert\") pod \"082744e7-08db-4d30-a96f-1d198b8cde90\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " Sep 08 14:18:38.292133 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.292052 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/082744e7-08db-4d30-a96f-1d198b8cde90-console-oauth-config\") pod \"082744e7-08db-4d30-a96f-1d198b8cde90\" (UID: \"082744e7-08db-4d30-a96f-1d198b8cde90\") " Sep 08 14:18:38.292401 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.292296 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "082744e7-08db-4d30-a96f-1d198b8cde90" (UID: "082744e7-08db-4d30-a96f-1d198b8cde90"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:18:38.292453 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.292434 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "082744e7-08db-4d30-a96f-1d198b8cde90" (UID: "082744e7-08db-4d30-a96f-1d198b8cde90"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:18:38.292495 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.292441 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-console-config" (OuterVolumeSpecName: "console-config") pod "082744e7-08db-4d30-a96f-1d198b8cde90" (UID: "082744e7-08db-4d30-a96f-1d198b8cde90"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:18:38.292495 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.292454 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-service-ca" (OuterVolumeSpecName: "service-ca") pod "082744e7-08db-4d30-a96f-1d198b8cde90" (UID: "082744e7-08db-4d30-a96f-1d198b8cde90"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:18:38.294319 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.294290 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082744e7-08db-4d30-a96f-1d198b8cde90-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "082744e7-08db-4d30-a96f-1d198b8cde90" (UID: "082744e7-08db-4d30-a96f-1d198b8cde90"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:18:38.294455 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.294439 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082744e7-08db-4d30-a96f-1d198b8cde90-kube-api-access-npqkd" (OuterVolumeSpecName: "kube-api-access-npqkd") pod "082744e7-08db-4d30-a96f-1d198b8cde90" (UID: "082744e7-08db-4d30-a96f-1d198b8cde90"). InnerVolumeSpecName "kube-api-access-npqkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:18:38.294505 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.294469 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082744e7-08db-4d30-a96f-1d198b8cde90-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "082744e7-08db-4d30-a96f-1d198b8cde90" (UID: "082744e7-08db-4d30-a96f-1d198b8cde90"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:18:38.393450 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.393409 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/082744e7-08db-4d30-a96f-1d198b8cde90-console-oauth-config\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:38.393450 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.393442 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-trusted-ca-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:38.393450 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.393451 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-console-config\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:38.393450 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.393460 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npqkd\" (UniqueName: \"kubernetes.io/projected/082744e7-08db-4d30-a96f-1d198b8cde90-kube-api-access-npqkd\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:38.393700 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.393469 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/082744e7-08db-4d30-a96f-1d198b8cde90-console-serving-cert\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:38.393700 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.393479 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:38.393700 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:38.393486 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/082744e7-08db-4d30-a96f-1d198b8cde90-oauth-serving-cert\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:18:39.017471 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:39.017439 2578 generic.go:356] "Generic (PLEG): container finished" podID="082744e7-08db-4d30-a96f-1d198b8cde90" containerID="7981730b129e566fff411fcd0ee24f7403d3a87facef698d3f5fba7dc425b902" exitCode=0 Sep 08 14:18:39.017901 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:39.017491 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9c4545c5-6zzwc" Sep 08 14:18:39.017901 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:39.017538 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9c4545c5-6zzwc" event={"ID":"082744e7-08db-4d30-a96f-1d198b8cde90","Type":"ContainerDied","Data":"7981730b129e566fff411fcd0ee24f7403d3a87facef698d3f5fba7dc425b902"} Sep 08 14:18:39.017901 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:39.017593 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9c4545c5-6zzwc" event={"ID":"082744e7-08db-4d30-a96f-1d198b8cde90","Type":"ContainerDied","Data":"3e56041d5624d9ebf7177529a4369d9aa215440522fdf5502c1154dd1182c990"} Sep 08 14:18:39.017901 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:39.017616 2578 scope.go:117] "RemoveContainer" containerID="7981730b129e566fff411fcd0ee24f7403d3a87facef698d3f5fba7dc425b902" Sep 08 14:18:39.026693 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:39.026678 2578 scope.go:117] "RemoveContainer" containerID="7981730b129e566fff411fcd0ee24f7403d3a87facef698d3f5fba7dc425b902" Sep 08 14:18:39.026958 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:18:39.026941 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7981730b129e566fff411fcd0ee24f7403d3a87facef698d3f5fba7dc425b902\": container with ID starting with 7981730b129e566fff411fcd0ee24f7403d3a87facef698d3f5fba7dc425b902 not found: ID does not exist" containerID="7981730b129e566fff411fcd0ee24f7403d3a87facef698d3f5fba7dc425b902" Sep 08 14:18:39.027004 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:39.026969 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7981730b129e566fff411fcd0ee24f7403d3a87facef698d3f5fba7dc425b902"} err="failed to get container status \"7981730b129e566fff411fcd0ee24f7403d3a87facef698d3f5fba7dc425b902\": rpc error: code = NotFound desc = could not find container \"7981730b129e566fff411fcd0ee24f7403d3a87facef698d3f5fba7dc425b902\": container with ID starting with 7981730b129e566fff411fcd0ee24f7403d3a87facef698d3f5fba7dc425b902 not found: ID does not exist" Sep 08 14:18:39.038744 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:39.038708 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d9c4545c5-6zzwc"] Sep 08 14:18:39.045438 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:39.045417 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d9c4545c5-6zzwc"] Sep 08 14:18:40.240276 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:18:40.240241 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="082744e7-08db-4d30-a96f-1d198b8cde90" path="/var/lib/kubelet/pods/082744e7-08db-4d30-a96f-1d198b8cde90/volumes" Sep 08 14:19:19.150074 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.150006 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7dbfdccd47-kwgls"] Sep 08 14:19:19.150473 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.150334 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2149aba-5979-43c9-aba4-6c698d36eb86" containerName="console" Sep 08 14:19:19.150473 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.150348 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2149aba-5979-43c9-aba4-6c698d36eb86" containerName="console" Sep 08 14:19:19.150473 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.150372 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="082744e7-08db-4d30-a96f-1d198b8cde90" containerName="console" Sep 08 14:19:19.150473 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.150377 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="082744e7-08db-4d30-a96f-1d198b8cde90" containerName="console" Sep 08 14:19:19.150473 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.150424 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="082744e7-08db-4d30-a96f-1d198b8cde90" containerName="console" Sep 08 14:19:19.150473 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.150431 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2149aba-5979-43c9-aba4-6c698d36eb86" containerName="console" Sep 08 14:19:19.153253 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.153232 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.164521 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.164490 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dbfdccd47-kwgls"] Sep 08 14:19:19.245398 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.245361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-serving-cert\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.245398 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.245399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-oauth-config\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.245648 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.245421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-config\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.245648 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.245502 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-trusted-ca-bundle\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.245648 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.245544 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-service-ca\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.245648 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.245559 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzrrn\" (UniqueName: \"kubernetes.io/projected/2b4b13d3-2198-4062-9486-d31df6aa50a8-kube-api-access-nzrrn\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.245648 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.245579 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-oauth-serving-cert\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.346896 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.346860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-service-ca\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.346896 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.346895 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzrrn\" (UniqueName: \"kubernetes.io/projected/2b4b13d3-2198-4062-9486-d31df6aa50a8-kube-api-access-nzrrn\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.347072 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.346916 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-oauth-serving-cert\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.347072 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.347039 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-serving-cert\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.347147 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.347077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-oauth-config\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.347147 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.347099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-config\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.347224 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.347167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-trusted-ca-bundle\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.347692 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.347657 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-service-ca\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.347832 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.347661 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-oauth-serving-cert\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.347938 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.347922 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-trusted-ca-bundle\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.348034 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.348004 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-config\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.349768 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.349717 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-oauth-config\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.349858 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.349819 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-serving-cert\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.357304 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.357282 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzrrn\" (UniqueName: \"kubernetes.io/projected/2b4b13d3-2198-4062-9486-d31df6aa50a8-kube-api-access-nzrrn\") pod \"console-7dbfdccd47-kwgls\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.463494 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.463462 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:19.737159 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:19.737083 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7dbfdccd47-kwgls"] Sep 08 14:19:19.740382 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:19:19.740355 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b4b13d3_2198_4062_9486_d31df6aa50a8.slice/crio-336f48cec137ee7921311e775cbaee5cdb22e45926ff2e610c988e275f688abd WatchSource:0}: Error finding container 336f48cec137ee7921311e775cbaee5cdb22e45926ff2e610c988e275f688abd: Status 404 returned error can't find the container with id 336f48cec137ee7921311e775cbaee5cdb22e45926ff2e610c988e275f688abd Sep 08 14:19:20.196379 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:20.196291 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbfdccd47-kwgls" event={"ID":"2b4b13d3-2198-4062-9486-d31df6aa50a8","Type":"ContainerStarted","Data":"1c4cdcf9cf99946150529cb68dc0fa4424393df564bd477f8c4278e73b9faba6"} Sep 08 14:19:20.196379 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:20.196355 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbfdccd47-kwgls" event={"ID":"2b4b13d3-2198-4062-9486-d31df6aa50a8","Type":"ContainerStarted","Data":"336f48cec137ee7921311e775cbaee5cdb22e45926ff2e610c988e275f688abd"} Sep 08 14:19:20.217945 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:20.217888 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7dbfdccd47-kwgls" podStartSLOduration=1.217873285 podStartE2EDuration="1.217873285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:19:20.216043682 +0000 UTC m=+172.500360824" watchObservedRunningTime="2026-09-08 14:19:20.217873285 +0000 UTC m=+172.502190399" Sep 08 14:19:29.464373 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:29.464339 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:29.464832 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:29.464436 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:29.469066 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:29.469047 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:30.242233 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:30.242204 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:19:30.289862 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:30.289814 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ddf646b7f-74fz8"] Sep 08 14:19:55.314278 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.314221 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-ddf646b7f-74fz8" podUID="b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0" containerName="console" containerID="cri-o://8c2ae8fab951cd4d82402ed8c56f7a1ba1580b80132a3d57536aecf7f5725dd7" gracePeriod=15 Sep 08 14:19:55.557580 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.557556 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:19:55.647017 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.646927 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-oauth-serving-cert\") pod \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " Sep 08 14:19:55.647017 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.646967 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-service-ca\") pod \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " Sep 08 14:19:55.647017 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.646997 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-oauth-config\") pod \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " Sep 08 14:19:55.647017 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.647018 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-config\") pod \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " Sep 08 14:19:55.647340 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.647057 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pszg\" (UniqueName: \"kubernetes.io/projected/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-kube-api-access-2pszg\") pod \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " Sep 08 14:19:55.647340 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.647072 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-trusted-ca-bundle\") pod \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " Sep 08 14:19:55.647340 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.647145 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-serving-cert\") pod \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\" (UID: \"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0\") " Sep 08 14:19:55.647487 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.647330 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0" (UID: "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:19:55.647562 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.647537 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-service-ca" (OuterVolumeSpecName: "service-ca") pod "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0" (UID: "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:19:55.647619 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.647556 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0" (UID: "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:19:55.647682 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.647639 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-config" (OuterVolumeSpecName: "console-config") pod "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0" (UID: "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:19:55.649418 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.649388 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0" (UID: "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:19:55.649527 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.649434 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0" (UID: "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:19:55.649527 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.649448 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-kube-api-access-2pszg" (OuterVolumeSpecName: "kube-api-access-2pszg") pod "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0" (UID: "b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0"). InnerVolumeSpecName "kube-api-access-2pszg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:19:55.748474 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.748436 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-serving-cert\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:19:55.748474 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.748466 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-oauth-serving-cert\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:19:55.748474 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.748483 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:19:55.748699 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.748493 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-oauth-config\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:19:55.748699 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.748502 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-console-config\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:19:55.748699 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.748509 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pszg\" (UniqueName: \"kubernetes.io/projected/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-kube-api-access-2pszg\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:19:55.748699 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:55.748518 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0-trusted-ca-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:19:56.340671 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:56.340641 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0" containerID="8c2ae8fab951cd4d82402ed8c56f7a1ba1580b80132a3d57536aecf7f5725dd7" exitCode=0 Sep 08 14:19:56.341051 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:56.340688 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddf646b7f-74fz8" Sep 08 14:19:56.341051 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:56.340739 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddf646b7f-74fz8" event={"ID":"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0","Type":"ContainerDied","Data":"8c2ae8fab951cd4d82402ed8c56f7a1ba1580b80132a3d57536aecf7f5725dd7"} Sep 08 14:19:56.341051 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:56.340787 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddf646b7f-74fz8" event={"ID":"b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0","Type":"ContainerDied","Data":"79f82f25984435ea17a68a7c67604eac433edd057a30f229dbeb4a827fe72c74"} Sep 08 14:19:56.341051 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:56.340808 2578 scope.go:117] "RemoveContainer" containerID="8c2ae8fab951cd4d82402ed8c56f7a1ba1580b80132a3d57536aecf7f5725dd7" Sep 08 14:19:56.348989 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:56.348973 2578 scope.go:117] "RemoveContainer" containerID="8c2ae8fab951cd4d82402ed8c56f7a1ba1580b80132a3d57536aecf7f5725dd7" Sep 08 14:19:56.349252 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:19:56.349235 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2ae8fab951cd4d82402ed8c56f7a1ba1580b80132a3d57536aecf7f5725dd7\": container with ID starting with 8c2ae8fab951cd4d82402ed8c56f7a1ba1580b80132a3d57536aecf7f5725dd7 not found: ID does not exist" containerID="8c2ae8fab951cd4d82402ed8c56f7a1ba1580b80132a3d57536aecf7f5725dd7" Sep 08 14:19:56.349294 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:56.349258 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2ae8fab951cd4d82402ed8c56f7a1ba1580b80132a3d57536aecf7f5725dd7"} err="failed to get container status \"8c2ae8fab951cd4d82402ed8c56f7a1ba1580b80132a3d57536aecf7f5725dd7\": rpc error: code = NotFound desc = could not find container \"8c2ae8fab951cd4d82402ed8c56f7a1ba1580b80132a3d57536aecf7f5725dd7\": container with ID starting with 8c2ae8fab951cd4d82402ed8c56f7a1ba1580b80132a3d57536aecf7f5725dd7 not found: ID does not exist" Sep 08 14:19:56.357115 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:56.357094 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ddf646b7f-74fz8"] Sep 08 14:19:56.360929 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:56.360909 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ddf646b7f-74fz8"] Sep 08 14:19:58.240543 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:19:58.240514 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0" path="/var/lib/kubelet/pods/b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0/volumes" Sep 08 14:21:09.694848 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.694817 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc"] Sep 08 14:21:09.695380 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.695258 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0" containerName="console" Sep 08 14:21:09.695380 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.695275 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0" containerName="console" Sep 08 14:21:09.695495 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.695381 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6b880e6-3b0b-4a55-8b41-bd8189a4f1f0" containerName="console" Sep 08 14:21:09.698491 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.698470 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" Sep 08 14:21:09.702693 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.702664 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 14:21:09.702832 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.702764 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-snnvk\"" Sep 08 14:21:09.703092 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.703077 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 14:21:09.716332 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.716301 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc"] Sep 08 14:21:09.771985 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.771953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb8b6064-0858-44ba-bed5-12d4313c3c6d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc\" (UID: \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" Sep 08 14:21:09.772134 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.772022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltrst\" (UniqueName: \"kubernetes.io/projected/bb8b6064-0858-44ba-bed5-12d4313c3c6d-kube-api-access-ltrst\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc\" (UID: \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" Sep 08 14:21:09.772134 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.772059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb8b6064-0858-44ba-bed5-12d4313c3c6d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc\" (UID: \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" Sep 08 14:21:09.872966 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.872935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb8b6064-0858-44ba-bed5-12d4313c3c6d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc\" (UID: \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" Sep 08 14:21:09.873138 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.872989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltrst\" (UniqueName: \"kubernetes.io/projected/bb8b6064-0858-44ba-bed5-12d4313c3c6d-kube-api-access-ltrst\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc\" (UID: \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" Sep 08 14:21:09.873138 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.873016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb8b6064-0858-44ba-bed5-12d4313c3c6d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc\" (UID: \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" Sep 08 14:21:09.873393 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.873375 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb8b6064-0858-44ba-bed5-12d4313c3c6d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc\" (UID: \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" Sep 08 14:21:09.873430 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.873379 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb8b6064-0858-44ba-bed5-12d4313c3c6d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc\" (UID: \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" Sep 08 14:21:09.882150 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:09.882122 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltrst\" (UniqueName: \"kubernetes.io/projected/bb8b6064-0858-44ba-bed5-12d4313c3c6d-kube-api-access-ltrst\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc\" (UID: \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" Sep 08 14:21:10.007997 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:10.007959 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" Sep 08 14:21:10.280258 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:10.280176 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc"] Sep 08 14:21:10.283175 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:21:10.283149 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8b6064_0858_44ba_bed5_12d4313c3c6d.slice/crio-6de93aefbd1a9fa66d36b04b055333fb98a575fc0ec104296f4bf36f96e09a0d WatchSource:0}: Error finding container 6de93aefbd1a9fa66d36b04b055333fb98a575fc0ec104296f4bf36f96e09a0d: Status 404 returned error can't find the container with id 6de93aefbd1a9fa66d36b04b055333fb98a575fc0ec104296f4bf36f96e09a0d Sep 08 14:21:10.629470 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:10.629387 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" event={"ID":"bb8b6064-0858-44ba-bed5-12d4313c3c6d","Type":"ContainerStarted","Data":"6de93aefbd1a9fa66d36b04b055333fb98a575fc0ec104296f4bf36f96e09a0d"} Sep 08 14:21:16.659243 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:16.659207 2578 generic.go:356] "Generic (PLEG): container finished" podID="bb8b6064-0858-44ba-bed5-12d4313c3c6d" containerID="9bc1b02e2012251cf4fb442335254383e3ade2ca4869aa6eba3622fe0fb49eaf" exitCode=0 Sep 08 14:21:16.659664 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:16.659287 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" event={"ID":"bb8b6064-0858-44ba-bed5-12d4313c3c6d","Type":"ContainerDied","Data":"9bc1b02e2012251cf4fb442335254383e3ade2ca4869aa6eba3622fe0fb49eaf"} Sep 08 14:21:19.673985 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:19.673942 2578 generic.go:356] "Generic (PLEG): container finished" podID="bb8b6064-0858-44ba-bed5-12d4313c3c6d" containerID="68e88dc4dbb2ec87fb6959e577710f60b124b8493e674cee45e53c2702dc8886" exitCode=0 Sep 08 14:21:19.674368 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:19.674020 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" event={"ID":"bb8b6064-0858-44ba-bed5-12d4313c3c6d","Type":"ContainerDied","Data":"68e88dc4dbb2ec87fb6959e577710f60b124b8493e674cee45e53c2702dc8886"} Sep 08 14:21:26.704923 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:26.704813 2578 generic.go:356] "Generic (PLEG): container finished" podID="bb8b6064-0858-44ba-bed5-12d4313c3c6d" containerID="4875bb30cfd4f4b949e693f0cd820f0ba2f5a6c10b380fcd55a695601882dc5e" exitCode=0 Sep 08 14:21:26.704923 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:26.704867 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" event={"ID":"bb8b6064-0858-44ba-bed5-12d4313c3c6d","Type":"ContainerDied","Data":"4875bb30cfd4f4b949e693f0cd820f0ba2f5a6c10b380fcd55a695601882dc5e"} Sep 08 14:21:27.835909 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:27.835886 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" Sep 08 14:21:27.933019 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:27.932987 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltrst\" (UniqueName: \"kubernetes.io/projected/bb8b6064-0858-44ba-bed5-12d4313c3c6d-kube-api-access-ltrst\") pod \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\" (UID: \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\") " Sep 08 14:21:27.933150 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:27.933048 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb8b6064-0858-44ba-bed5-12d4313c3c6d-bundle\") pod \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\" (UID: \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\") " Sep 08 14:21:27.933231 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:27.933213 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb8b6064-0858-44ba-bed5-12d4313c3c6d-util\") pod \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\" (UID: \"bb8b6064-0858-44ba-bed5-12d4313c3c6d\") " Sep 08 14:21:27.933659 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:27.933621 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8b6064-0858-44ba-bed5-12d4313c3c6d-bundle" (OuterVolumeSpecName: "bundle") pod "bb8b6064-0858-44ba-bed5-12d4313c3c6d" (UID: "bb8b6064-0858-44ba-bed5-12d4313c3c6d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:21:27.935302 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:27.935278 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8b6064-0858-44ba-bed5-12d4313c3c6d-kube-api-access-ltrst" (OuterVolumeSpecName: "kube-api-access-ltrst") pod "bb8b6064-0858-44ba-bed5-12d4313c3c6d" (UID: "bb8b6064-0858-44ba-bed5-12d4313c3c6d"). InnerVolumeSpecName "kube-api-access-ltrst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:21:27.938585 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:27.938562 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb8b6064-0858-44ba-bed5-12d4313c3c6d-util" (OuterVolumeSpecName: "util") pod "bb8b6064-0858-44ba-bed5-12d4313c3c6d" (UID: "bb8b6064-0858-44ba-bed5-12d4313c3c6d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:21:28.034302 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:28.034244 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltrst\" (UniqueName: \"kubernetes.io/projected/bb8b6064-0858-44ba-bed5-12d4313c3c6d-kube-api-access-ltrst\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:21:28.034302 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:28.034268 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bb8b6064-0858-44ba-bed5-12d4313c3c6d-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:21:28.034302 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:28.034277 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bb8b6064-0858-44ba-bed5-12d4313c3c6d-util\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:21:28.116514 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:28.116488 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:21:28.119372 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:28.119350 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:21:28.124492 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:28.124475 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:21:28.126996 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:28.126980 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:21:28.127584 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:28.127565 2578 kubelet.go:1648] "Image garbage collection succeeded" Sep 08 14:21:28.715860 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:28.715833 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" Sep 08 14:21:28.716050 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:28.715857 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mtgpc" event={"ID":"bb8b6064-0858-44ba-bed5-12d4313c3c6d","Type":"ContainerDied","Data":"6de93aefbd1a9fa66d36b04b055333fb98a575fc0ec104296f4bf36f96e09a0d"} Sep 08 14:21:28.716050 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:28.715904 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6de93aefbd1a9fa66d36b04b055333fb98a575fc0ec104296f4bf36f96e09a0d" Sep 08 14:21:37.027656 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.027620 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r"] Sep 08 14:21:37.028117 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.027953 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8b6064-0858-44ba-bed5-12d4313c3c6d" containerName="extract" Sep 08 14:21:37.028117 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.027965 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8b6064-0858-44ba-bed5-12d4313c3c6d" containerName="extract" Sep 08 14:21:37.028117 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.027979 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8b6064-0858-44ba-bed5-12d4313c3c6d" containerName="pull" Sep 08 14:21:37.028117 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.027984 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8b6064-0858-44ba-bed5-12d4313c3c6d" containerName="pull" Sep 08 14:21:37.028117 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.027995 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb8b6064-0858-44ba-bed5-12d4313c3c6d" containerName="util" Sep 08 14:21:37.028117 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.028001 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb8b6064-0858-44ba-bed5-12d4313c3c6d" containerName="util" Sep 08 14:21:37.028117 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.028054 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb8b6064-0858-44ba-bed5-12d4313c3c6d" containerName="extract" Sep 08 14:21:37.032763 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.032746 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" Sep 08 14:21:37.035041 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.035020 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 14:21:37.035705 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.035690 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-snnvk\"" Sep 08 14:21:37.035806 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.035701 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 14:21:37.042040 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.042020 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r"] Sep 08 14:21:37.108440 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.105686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38388339-fcad-441a-ba9d-8d62237d24c4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r\" (UID: \"38388339-fcad-441a-ba9d-8d62237d24c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" Sep 08 14:21:37.108440 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.105792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p696m\" (UniqueName: \"kubernetes.io/projected/38388339-fcad-441a-ba9d-8d62237d24c4-kube-api-access-p696m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r\" (UID: \"38388339-fcad-441a-ba9d-8d62237d24c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" Sep 08 14:21:37.108440 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.105829 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38388339-fcad-441a-ba9d-8d62237d24c4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r\" (UID: \"38388339-fcad-441a-ba9d-8d62237d24c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" Sep 08 14:21:37.206985 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.206951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p696m\" (UniqueName: \"kubernetes.io/projected/38388339-fcad-441a-ba9d-8d62237d24c4-kube-api-access-p696m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r\" (UID: \"38388339-fcad-441a-ba9d-8d62237d24c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" Sep 08 14:21:37.206985 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.206987 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38388339-fcad-441a-ba9d-8d62237d24c4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r\" (UID: \"38388339-fcad-441a-ba9d-8d62237d24c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" Sep 08 14:21:37.207186 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.207055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38388339-fcad-441a-ba9d-8d62237d24c4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r\" (UID: \"38388339-fcad-441a-ba9d-8d62237d24c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" Sep 08 14:21:37.207385 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.207370 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38388339-fcad-441a-ba9d-8d62237d24c4-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r\" (UID: \"38388339-fcad-441a-ba9d-8d62237d24c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" Sep 08 14:21:37.207422 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.207402 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38388339-fcad-441a-ba9d-8d62237d24c4-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r\" (UID: \"38388339-fcad-441a-ba9d-8d62237d24c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" Sep 08 14:21:37.216952 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.216927 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p696m\" (UniqueName: \"kubernetes.io/projected/38388339-fcad-441a-ba9d-8d62237d24c4-kube-api-access-p696m\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r\" (UID: \"38388339-fcad-441a-ba9d-8d62237d24c4\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" Sep 08 14:21:37.341463 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.341384 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" Sep 08 14:21:37.617148 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.617124 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r"] Sep 08 14:21:37.619033 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:21:37.619005 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38388339_fcad_441a_ba9d_8d62237d24c4.slice/crio-47ea82ea3c6cb1d33915c420a4a71d867d7215f2ecb4c5e14b2222ec2068660c WatchSource:0}: Error finding container 47ea82ea3c6cb1d33915c420a4a71d867d7215f2ecb4c5e14b2222ec2068660c: Status 404 returned error can't find the container with id 47ea82ea3c6cb1d33915c420a4a71d867d7215f2ecb4c5e14b2222ec2068660c Sep 08 14:21:37.751910 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.751870 2578 generic.go:356] "Generic (PLEG): container finished" podID="38388339-fcad-441a-ba9d-8d62237d24c4" containerID="0959ea86e5b41a93db39715064fee6fc28f14b2330961b1a252ae70932b2b5c4" exitCode=0 Sep 08 14:21:37.752017 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.751943 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" event={"ID":"38388339-fcad-441a-ba9d-8d62237d24c4","Type":"ContainerDied","Data":"0959ea86e5b41a93db39715064fee6fc28f14b2330961b1a252ae70932b2b5c4"} Sep 08 14:21:37.752017 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.751999 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" event={"ID":"38388339-fcad-441a-ba9d-8d62237d24c4","Type":"ContainerStarted","Data":"47ea82ea3c6cb1d33915c420a4a71d867d7215f2ecb4c5e14b2222ec2068660c"} Sep 08 14:21:37.753002 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:37.752986 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 14:21:38.744041 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:38.744006 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-s28x5"] Sep 08 14:21:38.746590 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:38.746570 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-s28x5" Sep 08 14:21:38.748972 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:38.748951 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 08 14:21:38.749092 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:38.749002 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 08 14:21:38.749706 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:38.749684 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-dh4jq\"" Sep 08 14:21:38.759917 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:38.759893 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-s28x5"] Sep 08 14:21:38.821422 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:38.821384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10487086-fd22-4597-9693-b01caceb1e10-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-s28x5\" (UID: \"10487086-fd22-4597-9693-b01caceb1e10\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s28x5" Sep 08 14:21:38.821580 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:38.821445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prbgh\" (UniqueName: \"kubernetes.io/projected/10487086-fd22-4597-9693-b01caceb1e10-kube-api-access-prbgh\") pod \"cert-manager-webhook-6486ffd997-s28x5\" (UID: \"10487086-fd22-4597-9693-b01caceb1e10\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s28x5" Sep 08 14:21:38.922023 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:38.921989 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prbgh\" (UniqueName: \"kubernetes.io/projected/10487086-fd22-4597-9693-b01caceb1e10-kube-api-access-prbgh\") pod \"cert-manager-webhook-6486ffd997-s28x5\" (UID: \"10487086-fd22-4597-9693-b01caceb1e10\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s28x5" Sep 08 14:21:38.922216 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:38.922104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10487086-fd22-4597-9693-b01caceb1e10-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-s28x5\" (UID: \"10487086-fd22-4597-9693-b01caceb1e10\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s28x5" Sep 08 14:21:38.931834 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:38.931804 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/10487086-fd22-4597-9693-b01caceb1e10-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-s28x5\" (UID: \"10487086-fd22-4597-9693-b01caceb1e10\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s28x5" Sep 08 14:21:38.932057 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:38.932038 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prbgh\" (UniqueName: \"kubernetes.io/projected/10487086-fd22-4597-9693-b01caceb1e10-kube-api-access-prbgh\") pod \"cert-manager-webhook-6486ffd997-s28x5\" (UID: \"10487086-fd22-4597-9693-b01caceb1e10\") " pod="cert-manager/cert-manager-webhook-6486ffd997-s28x5" Sep 08 14:21:39.065999 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:39.065920 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-s28x5" Sep 08 14:21:39.764451 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:39.764417 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" event={"ID":"38388339-fcad-441a-ba9d-8d62237d24c4","Type":"ContainerStarted","Data":"bb6a68688b5a709d0ab17b70e4679a9be959cdbebf97ed16b699ca47363a157c"} Sep 08 14:21:39.897465 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:39.897374 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-s28x5"] Sep 08 14:21:39.900061 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:21:39.900029 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10487086_fd22_4597_9693_b01caceb1e10.slice/crio-4ba2a08758e4c5e6cf239812258412d5af006f5de53b57ccad415793a5a4edde WatchSource:0}: Error finding container 4ba2a08758e4c5e6cf239812258412d5af006f5de53b57ccad415793a5a4edde: Status 404 returned error can't find the container with id 4ba2a08758e4c5e6cf239812258412d5af006f5de53b57ccad415793a5a4edde Sep 08 14:21:40.770422 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:40.770384 2578 generic.go:356] "Generic (PLEG): container finished" podID="38388339-fcad-441a-ba9d-8d62237d24c4" containerID="bb6a68688b5a709d0ab17b70e4679a9be959cdbebf97ed16b699ca47363a157c" exitCode=0 Sep 08 14:21:40.770876 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:40.770461 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" event={"ID":"38388339-fcad-441a-ba9d-8d62237d24c4","Type":"ContainerDied","Data":"bb6a68688b5a709d0ab17b70e4679a9be959cdbebf97ed16b699ca47363a157c"} Sep 08 14:21:40.771847 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:40.771825 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-s28x5" event={"ID":"10487086-fd22-4597-9693-b01caceb1e10","Type":"ContainerStarted","Data":"4ba2a08758e4c5e6cf239812258412d5af006f5de53b57ccad415793a5a4edde"} Sep 08 14:21:41.780163 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:41.780124 2578 generic.go:356] "Generic (PLEG): container finished" podID="38388339-fcad-441a-ba9d-8d62237d24c4" containerID="90dadb15527a45ebf3db850534efd8c999904ddb6044c8a5e75efcd1b5f2c90f" exitCode=0 Sep 08 14:21:41.780591 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:41.780266 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" event={"ID":"38388339-fcad-441a-ba9d-8d62237d24c4","Type":"ContainerDied","Data":"90dadb15527a45ebf3db850534efd8c999904ddb6044c8a5e75efcd1b5f2c90f"} Sep 08 14:21:42.921672 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:42.921648 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" Sep 08 14:21:42.957467 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:42.957441 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p696m\" (UniqueName: \"kubernetes.io/projected/38388339-fcad-441a-ba9d-8d62237d24c4-kube-api-access-p696m\") pod \"38388339-fcad-441a-ba9d-8d62237d24c4\" (UID: \"38388339-fcad-441a-ba9d-8d62237d24c4\") " Sep 08 14:21:42.957626 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:42.957490 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38388339-fcad-441a-ba9d-8d62237d24c4-bundle\") pod \"38388339-fcad-441a-ba9d-8d62237d24c4\" (UID: \"38388339-fcad-441a-ba9d-8d62237d24c4\") " Sep 08 14:21:42.957626 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:42.957523 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38388339-fcad-441a-ba9d-8d62237d24c4-util\") pod \"38388339-fcad-441a-ba9d-8d62237d24c4\" (UID: \"38388339-fcad-441a-ba9d-8d62237d24c4\") " Sep 08 14:21:42.957971 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:42.957928 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38388339-fcad-441a-ba9d-8d62237d24c4-bundle" (OuterVolumeSpecName: "bundle") pod "38388339-fcad-441a-ba9d-8d62237d24c4" (UID: "38388339-fcad-441a-ba9d-8d62237d24c4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:21:42.960212 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:42.960182 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38388339-fcad-441a-ba9d-8d62237d24c4-kube-api-access-p696m" (OuterVolumeSpecName: "kube-api-access-p696m") pod "38388339-fcad-441a-ba9d-8d62237d24c4" (UID: "38388339-fcad-441a-ba9d-8d62237d24c4"). InnerVolumeSpecName "kube-api-access-p696m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:21:42.963902 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:42.963876 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38388339-fcad-441a-ba9d-8d62237d24c4-util" (OuterVolumeSpecName: "util") pod "38388339-fcad-441a-ba9d-8d62237d24c4" (UID: "38388339-fcad-441a-ba9d-8d62237d24c4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:21:43.058573 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:43.058504 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p696m\" (UniqueName: \"kubernetes.io/projected/38388339-fcad-441a-ba9d-8d62237d24c4-kube-api-access-p696m\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:21:43.058573 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:43.058531 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38388339-fcad-441a-ba9d-8d62237d24c4-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:21:43.058573 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:43.058543 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38388339-fcad-441a-ba9d-8d62237d24c4-util\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:21:43.792633 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:43.792594 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-s28x5" event={"ID":"10487086-fd22-4597-9693-b01caceb1e10","Type":"ContainerStarted","Data":"a861569d4d5d76a54c116cd4ca230003593ad665ee5dd90dc262d454a87a83ed"} Sep 08 14:21:43.792859 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:43.792651 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-s28x5" Sep 08 14:21:43.794383 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:43.794363 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" event={"ID":"38388339-fcad-441a-ba9d-8d62237d24c4","Type":"ContainerDied","Data":"47ea82ea3c6cb1d33915c420a4a71d867d7215f2ecb4c5e14b2222ec2068660c"} Sep 08 14:21:43.794383 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:43.794375 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fqqs5r" Sep 08 14:21:43.794506 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:43.794398 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47ea82ea3c6cb1d33915c420a4a71d867d7215f2ecb4c5e14b2222ec2068660c" Sep 08 14:21:43.809444 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:43.809384 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-s28x5" podStartSLOduration=2.841123694 podStartE2EDuration="5.809367877s" totalImagesPullingTime="2.968244183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:21:43.809053212 +0000 UTC m=+316.093370339" watchObservedRunningTime="2026-09-08 14:21:43.809367877 +0000 UTC m=+316.093684996" Sep 08 14:21:46.862669 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:46.862639 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qh99h"] Sep 08 14:21:46.863020 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:46.862944 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38388339-fcad-441a-ba9d-8d62237d24c4" containerName="extract" Sep 08 14:21:46.863020 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:46.862956 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="38388339-fcad-441a-ba9d-8d62237d24c4" containerName="extract" Sep 08 14:21:46.863020 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:46.862965 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38388339-fcad-441a-ba9d-8d62237d24c4" containerName="pull" Sep 08 14:21:46.863020 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:46.862970 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="38388339-fcad-441a-ba9d-8d62237d24c4" containerName="pull" Sep 08 14:21:46.863020 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:46.862983 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38388339-fcad-441a-ba9d-8d62237d24c4" containerName="util" Sep 08 14:21:46.863020 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:46.862988 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="38388339-fcad-441a-ba9d-8d62237d24c4" containerName="util" Sep 08 14:21:46.863212 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:46.863050 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="38388339-fcad-441a-ba9d-8d62237d24c4" containerName="extract" Sep 08 14:21:46.864950 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:46.864935 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qh99h" Sep 08 14:21:46.867267 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:46.867242 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6fg2r\"" Sep 08 14:21:46.876207 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:46.876185 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qh99h"] Sep 08 14:21:46.992671 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:46.992638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn9hl\" (UniqueName: \"kubernetes.io/projected/68e4f72d-ec59-4935-a201-c28a9f156673-kube-api-access-qn9hl\") pod \"cert-manager-55b4d954d5-qh99h\" (UID: \"68e4f72d-ec59-4935-a201-c28a9f156673\") " pod="cert-manager/cert-manager-55b4d954d5-qh99h" Sep 08 14:21:46.992671 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:46.992675 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68e4f72d-ec59-4935-a201-c28a9f156673-bound-sa-token\") pod \"cert-manager-55b4d954d5-qh99h\" (UID: \"68e4f72d-ec59-4935-a201-c28a9f156673\") " pod="cert-manager/cert-manager-55b4d954d5-qh99h" Sep 08 14:21:47.093881 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:47.093850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn9hl\" (UniqueName: \"kubernetes.io/projected/68e4f72d-ec59-4935-a201-c28a9f156673-kube-api-access-qn9hl\") pod \"cert-manager-55b4d954d5-qh99h\" (UID: \"68e4f72d-ec59-4935-a201-c28a9f156673\") " pod="cert-manager/cert-manager-55b4d954d5-qh99h" Sep 08 14:21:47.093881 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:47.093881 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68e4f72d-ec59-4935-a201-c28a9f156673-bound-sa-token\") pod \"cert-manager-55b4d954d5-qh99h\" (UID: \"68e4f72d-ec59-4935-a201-c28a9f156673\") " pod="cert-manager/cert-manager-55b4d954d5-qh99h" Sep 08 14:21:47.104039 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:47.104011 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/68e4f72d-ec59-4935-a201-c28a9f156673-bound-sa-token\") pod \"cert-manager-55b4d954d5-qh99h\" (UID: \"68e4f72d-ec59-4935-a201-c28a9f156673\") " pod="cert-manager/cert-manager-55b4d954d5-qh99h" Sep 08 14:21:47.104209 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:47.104191 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn9hl\" (UniqueName: \"kubernetes.io/projected/68e4f72d-ec59-4935-a201-c28a9f156673-kube-api-access-qn9hl\") pod \"cert-manager-55b4d954d5-qh99h\" (UID: \"68e4f72d-ec59-4935-a201-c28a9f156673\") " pod="cert-manager/cert-manager-55b4d954d5-qh99h" Sep 08 14:21:47.173769 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:47.173694 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qh99h" Sep 08 14:21:47.443055 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:47.442958 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qh99h"] Sep 08 14:21:47.445648 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:21:47.445623 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68e4f72d_ec59_4935_a201_c28a9f156673.slice/crio-6b2b53d2cd6765628aa74b0d7438643e3ce14daed5a3d632bd65a87c8d3c49e6 WatchSource:0}: Error finding container 6b2b53d2cd6765628aa74b0d7438643e3ce14daed5a3d632bd65a87c8d3c49e6: Status 404 returned error can't find the container with id 6b2b53d2cd6765628aa74b0d7438643e3ce14daed5a3d632bd65a87c8d3c49e6 Sep 08 14:21:47.813112 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:47.813076 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qh99h" event={"ID":"68e4f72d-ec59-4935-a201-c28a9f156673","Type":"ContainerStarted","Data":"ee866cf64b23499059d902c8146202488be5913437962d6a21c30a3b456c8963"} Sep 08 14:21:47.813276 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:47.813117 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qh99h" event={"ID":"68e4f72d-ec59-4935-a201-c28a9f156673","Type":"ContainerStarted","Data":"6b2b53d2cd6765628aa74b0d7438643e3ce14daed5a3d632bd65a87c8d3c49e6"} Sep 08 14:21:47.830554 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:47.830505 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-qh99h" podStartSLOduration=1.8304930590000001 podStartE2EDuration="1.830493059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:21:47.829679378 +0000 UTC m=+320.113996493" watchObservedRunningTime="2026-09-08 14:21:47.830493059 +0000 UTC m=+320.114810173" Sep 08 14:21:49.801833 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:49.801803 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-s28x5" Sep 08 14:21:49.937114 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:49.937071 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk"] Sep 08 14:21:49.939423 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:49.939407 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk" Sep 08 14:21:49.941717 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:49.941689 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-hnlmq\"" Sep 08 14:21:49.941857 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:49.941692 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 08 14:21:49.941857 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:49.941834 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 08 14:21:49.948408 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:49.948373 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk"] Sep 08 14:21:50.020134 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:50.020098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18616aa8-b8a4-4959-84c8-22e91f878b43-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q2fqk\" (UID: \"18616aa8-b8a4-4959-84c8-22e91f878b43\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk" Sep 08 14:21:50.020285 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:50.020163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkhqw\" (UniqueName: \"kubernetes.io/projected/18616aa8-b8a4-4959-84c8-22e91f878b43-kube-api-access-pkhqw\") pod \"openshift-lws-operator-fd8ccff4c-q2fqk\" (UID: \"18616aa8-b8a4-4959-84c8-22e91f878b43\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk" Sep 08 14:21:50.121555 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:50.121464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18616aa8-b8a4-4959-84c8-22e91f878b43-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q2fqk\" (UID: \"18616aa8-b8a4-4959-84c8-22e91f878b43\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk" Sep 08 14:21:50.121555 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:50.121524 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkhqw\" (UniqueName: \"kubernetes.io/projected/18616aa8-b8a4-4959-84c8-22e91f878b43-kube-api-access-pkhqw\") pod \"openshift-lws-operator-fd8ccff4c-q2fqk\" (UID: \"18616aa8-b8a4-4959-84c8-22e91f878b43\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk" Sep 08 14:21:50.121911 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:50.121894 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/18616aa8-b8a4-4959-84c8-22e91f878b43-tmp\") pod \"openshift-lws-operator-fd8ccff4c-q2fqk\" (UID: \"18616aa8-b8a4-4959-84c8-22e91f878b43\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk" Sep 08 14:21:50.130840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:50.130810 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkhqw\" (UniqueName: \"kubernetes.io/projected/18616aa8-b8a4-4959-84c8-22e91f878b43-kube-api-access-pkhqw\") pod \"openshift-lws-operator-fd8ccff4c-q2fqk\" (UID: \"18616aa8-b8a4-4959-84c8-22e91f878b43\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk" Sep 08 14:21:50.248601 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:50.248568 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk" Sep 08 14:21:50.521867 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:50.521841 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk"] Sep 08 14:21:50.524497 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:21:50.524470 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18616aa8_b8a4_4959_84c8_22e91f878b43.slice/crio-1f6318e0018b860b0ecd29d6127f86c6539ef53a38483a1c0d69ab459f3aeeae WatchSource:0}: Error finding container 1f6318e0018b860b0ecd29d6127f86c6539ef53a38483a1c0d69ab459f3aeeae: Status 404 returned error can't find the container with id 1f6318e0018b860b0ecd29d6127f86c6539ef53a38483a1c0d69ab459f3aeeae Sep 08 14:21:50.828560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:50.828477 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk" event={"ID":"18616aa8-b8a4-4959-84c8-22e91f878b43","Type":"ContainerStarted","Data":"1f6318e0018b860b0ecd29d6127f86c6539ef53a38483a1c0d69ab459f3aeeae"} Sep 08 14:21:54.848477 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:54.848440 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk" event={"ID":"18616aa8-b8a4-4959-84c8-22e91f878b43","Type":"ContainerStarted","Data":"75ed53a696a703f978fc9d31d1bffc3ab50a200d99335786c8c1506824a077e3"} Sep 08 14:21:54.866604 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:21:54.866559 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-q2fqk" podStartSLOduration=2.352939329 podStartE2EDuration="5.866543984s" totalImagesPullingTime="3.513604655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:21:54.864779561 +0000 UTC m=+327.149096677" watchObservedRunningTime="2026-09-08 14:21:54.866543984 +0000 UTC m=+327.150861176" Sep 08 14:22:00.113542 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.113509 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz"] Sep 08 14:22:00.116168 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.116153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" Sep 08 14:22:00.118509 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.118480 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 14:22:00.118644 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.118529 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-snnvk\"" Sep 08 14:22:00.119266 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.119251 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 14:22:00.125896 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.125875 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz"] Sep 08 14:22:00.208092 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.208048 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4564fe07-f033-4ba0-9a08-c2d382e66587-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz\" (UID: \"4564fe07-f033-4ba0-9a08-c2d382e66587\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" Sep 08 14:22:00.208268 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.208134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl8h4\" (UniqueName: \"kubernetes.io/projected/4564fe07-f033-4ba0-9a08-c2d382e66587-kube-api-access-gl8h4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz\" (UID: \"4564fe07-f033-4ba0-9a08-c2d382e66587\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" Sep 08 14:22:00.208268 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.208186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4564fe07-f033-4ba0-9a08-c2d382e66587-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz\" (UID: \"4564fe07-f033-4ba0-9a08-c2d382e66587\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" Sep 08 14:22:00.309146 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.309094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4564fe07-f033-4ba0-9a08-c2d382e66587-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz\" (UID: \"4564fe07-f033-4ba0-9a08-c2d382e66587\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" Sep 08 14:22:00.309339 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.309164 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4564fe07-f033-4ba0-9a08-c2d382e66587-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz\" (UID: \"4564fe07-f033-4ba0-9a08-c2d382e66587\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" Sep 08 14:22:00.309339 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.309260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl8h4\" (UniqueName: \"kubernetes.io/projected/4564fe07-f033-4ba0-9a08-c2d382e66587-kube-api-access-gl8h4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz\" (UID: \"4564fe07-f033-4ba0-9a08-c2d382e66587\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" Sep 08 14:22:00.309559 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.309535 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4564fe07-f033-4ba0-9a08-c2d382e66587-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz\" (UID: \"4564fe07-f033-4ba0-9a08-c2d382e66587\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" Sep 08 14:22:00.309620 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.309543 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4564fe07-f033-4ba0-9a08-c2d382e66587-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz\" (UID: \"4564fe07-f033-4ba0-9a08-c2d382e66587\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" Sep 08 14:22:00.321837 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.321804 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl8h4\" (UniqueName: \"kubernetes.io/projected/4564fe07-f033-4ba0-9a08-c2d382e66587-kube-api-access-gl8h4\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz\" (UID: \"4564fe07-f033-4ba0-9a08-c2d382e66587\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" Sep 08 14:22:00.425802 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.425720 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" Sep 08 14:22:00.699708 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.699679 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz"] Sep 08 14:22:00.704192 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:22:00.704162 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4564fe07_f033_4ba0_9a08_c2d382e66587.slice/crio-98f635b2e04719e786f208214f26d59c40f347fc2b58d9c3a1f459d3c103b036 WatchSource:0}: Error finding container 98f635b2e04719e786f208214f26d59c40f347fc2b58d9c3a1f459d3c103b036: Status 404 returned error can't find the container with id 98f635b2e04719e786f208214f26d59c40f347fc2b58d9c3a1f459d3c103b036 Sep 08 14:22:00.876595 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.876557 2578 generic.go:356] "Generic (PLEG): container finished" podID="4564fe07-f033-4ba0-9a08-c2d382e66587" containerID="23a378aad1849258e6854b9c82283092845876407edddc5ccfd5838e388785d3" exitCode=0 Sep 08 14:22:00.876763 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.876601 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" event={"ID":"4564fe07-f033-4ba0-9a08-c2d382e66587","Type":"ContainerDied","Data":"23a378aad1849258e6854b9c82283092845876407edddc5ccfd5838e388785d3"} Sep 08 14:22:00.876763 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:00.876656 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" event={"ID":"4564fe07-f033-4ba0-9a08-c2d382e66587","Type":"ContainerStarted","Data":"98f635b2e04719e786f208214f26d59c40f347fc2b58d9c3a1f459d3c103b036"} Sep 08 14:22:01.884286 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:01.884183 2578 generic.go:356] "Generic (PLEG): container finished" podID="4564fe07-f033-4ba0-9a08-c2d382e66587" containerID="390c6d41092769ca94da796fa24c2ce7c9ca06d067ee4374399f955b09cd15a3" exitCode=0 Sep 08 14:22:01.884286 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:01.884224 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" event={"ID":"4564fe07-f033-4ba0-9a08-c2d382e66587","Type":"ContainerDied","Data":"390c6d41092769ca94da796fa24c2ce7c9ca06d067ee4374399f955b09cd15a3"} Sep 08 14:22:02.891107 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:02.891068 2578 generic.go:356] "Generic (PLEG): container finished" podID="4564fe07-f033-4ba0-9a08-c2d382e66587" containerID="c564c261a86534f4aff1d1344155730777a1599a116ab214c5fb4a368129b725" exitCode=0 Sep 08 14:22:02.891542 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:02.891166 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" event={"ID":"4564fe07-f033-4ba0-9a08-c2d382e66587","Type":"ContainerDied","Data":"c564c261a86534f4aff1d1344155730777a1599a116ab214c5fb4a368129b725"} Sep 08 14:22:04.026045 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:04.026022 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" Sep 08 14:22:04.143430 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:04.143393 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gl8h4\" (UniqueName: \"kubernetes.io/projected/4564fe07-f033-4ba0-9a08-c2d382e66587-kube-api-access-gl8h4\") pod \"4564fe07-f033-4ba0-9a08-c2d382e66587\" (UID: \"4564fe07-f033-4ba0-9a08-c2d382e66587\") " Sep 08 14:22:04.143596 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:04.143465 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4564fe07-f033-4ba0-9a08-c2d382e66587-bundle\") pod \"4564fe07-f033-4ba0-9a08-c2d382e66587\" (UID: \"4564fe07-f033-4ba0-9a08-c2d382e66587\") " Sep 08 14:22:04.143596 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:04.143516 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4564fe07-f033-4ba0-9a08-c2d382e66587-util\") pod \"4564fe07-f033-4ba0-9a08-c2d382e66587\" (UID: \"4564fe07-f033-4ba0-9a08-c2d382e66587\") " Sep 08 14:22:04.144176 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:04.144143 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4564fe07-f033-4ba0-9a08-c2d382e66587-bundle" (OuterVolumeSpecName: "bundle") pod "4564fe07-f033-4ba0-9a08-c2d382e66587" (UID: "4564fe07-f033-4ba0-9a08-c2d382e66587"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:22:04.145617 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:04.145592 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4564fe07-f033-4ba0-9a08-c2d382e66587-kube-api-access-gl8h4" (OuterVolumeSpecName: "kube-api-access-gl8h4") pod "4564fe07-f033-4ba0-9a08-c2d382e66587" (UID: "4564fe07-f033-4ba0-9a08-c2d382e66587"). InnerVolumeSpecName "kube-api-access-gl8h4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:22:04.148887 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:04.148846 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4564fe07-f033-4ba0-9a08-c2d382e66587-util" (OuterVolumeSpecName: "util") pod "4564fe07-f033-4ba0-9a08-c2d382e66587" (UID: "4564fe07-f033-4ba0-9a08-c2d382e66587"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:22:04.244126 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:04.244099 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl8h4\" (UniqueName: \"kubernetes.io/projected/4564fe07-f033-4ba0-9a08-c2d382e66587-kube-api-access-gl8h4\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:22:04.244126 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:04.244125 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4564fe07-f033-4ba0-9a08-c2d382e66587-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:22:04.244126 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:04.244136 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4564fe07-f033-4ba0-9a08-c2d382e66587-util\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:22:04.909261 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:04.909226 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" event={"ID":"4564fe07-f033-4ba0-9a08-c2d382e66587","Type":"ContainerDied","Data":"98f635b2e04719e786f208214f26d59c40f347fc2b58d9c3a1f459d3c103b036"} Sep 08 14:22:04.909458 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:04.909272 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98f635b2e04719e786f208214f26d59c40f347fc2b58d9c3a1f459d3c103b036" Sep 08 14:22:04.909458 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:04.909271 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afr4tvz" Sep 08 14:22:10.372466 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.372429 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q"] Sep 08 14:22:10.372920 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.372742 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4564fe07-f033-4ba0-9a08-c2d382e66587" containerName="pull" Sep 08 14:22:10.372920 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.372755 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4564fe07-f033-4ba0-9a08-c2d382e66587" containerName="pull" Sep 08 14:22:10.372920 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.372769 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4564fe07-f033-4ba0-9a08-c2d382e66587" containerName="extract" Sep 08 14:22:10.372920 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.372775 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4564fe07-f033-4ba0-9a08-c2d382e66587" containerName="extract" Sep 08 14:22:10.372920 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.372789 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4564fe07-f033-4ba0-9a08-c2d382e66587" containerName="util" Sep 08 14:22:10.372920 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.372803 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4564fe07-f033-4ba0-9a08-c2d382e66587" containerName="util" Sep 08 14:22:10.372920 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.372871 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4564fe07-f033-4ba0-9a08-c2d382e66587" containerName="extract" Sep 08 14:22:10.377023 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.377008 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" Sep 08 14:22:10.379207 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.379182 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 14:22:10.379971 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.379953 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 14:22:10.380067 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.380003 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-snnvk\"" Sep 08 14:22:10.385224 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.385200 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q"] Sep 08 14:22:10.495188 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.495151 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsf6c\" (UniqueName: \"kubernetes.io/projected/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-kube-api-access-vsf6c\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q\" (UID: \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" Sep 08 14:22:10.495367 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.495198 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q\" (UID: \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" Sep 08 14:22:10.495367 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.495315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q\" (UID: \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" Sep 08 14:22:10.596610 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.596571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q\" (UID: \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" Sep 08 14:22:10.596784 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.596620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsf6c\" (UniqueName: \"kubernetes.io/projected/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-kube-api-access-vsf6c\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q\" (UID: \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" Sep 08 14:22:10.596784 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.596651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q\" (UID: \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" Sep 08 14:22:10.596997 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.596975 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q\" (UID: \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" Sep 08 14:22:10.597037 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.596994 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q\" (UID: \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" Sep 08 14:22:10.609106 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.609074 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsf6c\" (UniqueName: \"kubernetes.io/projected/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-kube-api-access-vsf6c\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q\" (UID: \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" Sep 08 14:22:10.686519 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.686441 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" Sep 08 14:22:10.964258 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:10.964155 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q"] Sep 08 14:22:10.966431 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:22:10.966404 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3880ac1a_3eb6_4f74_b1d1_cbe6d5aab050.slice/crio-62e1a9d938df78947292be71ec90c91efc9d864d892359df770922addd152e69 WatchSource:0}: Error finding container 62e1a9d938df78947292be71ec90c91efc9d864d892359df770922addd152e69: Status 404 returned error can't find the container with id 62e1a9d938df78947292be71ec90c91efc9d864d892359df770922addd152e69 Sep 08 14:22:11.942151 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:11.942063 2578 generic.go:356] "Generic (PLEG): container finished" podID="3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050" containerID="6986852e57f148aac03e62bbb4407f202226d6efc4a6e31b7bf175ef441b0dc6" exitCode=0 Sep 08 14:22:11.942585 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:11.942154 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" event={"ID":"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050","Type":"ContainerDied","Data":"6986852e57f148aac03e62bbb4407f202226d6efc4a6e31b7bf175ef441b0dc6"} Sep 08 14:22:11.942585 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:11.942208 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" event={"ID":"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050","Type":"ContainerStarted","Data":"62e1a9d938df78947292be71ec90c91efc9d864d892359df770922addd152e69"} Sep 08 14:22:12.853545 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:12.853511 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv"] Sep 08 14:22:12.856787 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:12.856771 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:12.859041 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:12.859015 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 08 14:22:12.859041 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:12.859014 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 08 14:22:12.859243 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:12.859076 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-js9cr\"" Sep 08 14:22:12.859243 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:12.859125 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 08 14:22:12.859243 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:12.859142 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 08 14:22:12.867831 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:12.867809 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv"] Sep 08 14:22:12.948772 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:12.948656 2578 generic.go:356] "Generic (PLEG): container finished" podID="3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050" containerID="b82c044d9401640ed75bd9a1910142389b22b82c2cf4535b85da42d1825fdedf" exitCode=0 Sep 08 14:22:12.948772 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:12.948697 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" event={"ID":"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050","Type":"ContainerDied","Data":"b82c044d9401640ed75bd9a1910142389b22b82c2cf4535b85da42d1825fdedf"} Sep 08 14:22:13.017273 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.017245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kr5r\" (UniqueName: \"kubernetes.io/projected/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-kube-api-access-4kr5r\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.017401 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.017342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-tmp\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.017401 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.017370 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.017401 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.017396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-manifests\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.017523 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.017474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-webhook-cert\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.118516 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.118489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-manifests\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.118634 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.118530 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-webhook-cert\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.118634 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.118579 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kr5r\" (UniqueName: \"kubernetes.io/projected/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-kube-api-access-4kr5r\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.118787 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.118768 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-tmp\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.118841 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.118804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.118941 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.118920 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-manifests\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.120928 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.120900 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-tmp\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.121058 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.121043 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-webhook-cert\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.121104 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.121079 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.127752 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.127717 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kr5r\" (UniqueName: \"kubernetes.io/projected/7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9-kube-api-access-4kr5r\") pod \"opendatahub-operator-controller-manager-566d57d45b-gwwdv\" (UID: \"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9\") " pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.166810 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.166782 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:13.443613 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.443567 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv"] Sep 08 14:22:13.445855 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:22:13.445831 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7510c360_4e6f_4dbf_a68c_9f51ec6f0bd9.slice/crio-c0165edc9b112d0cedb920020a46938ee3cbb3ee35e833af8838871c7a1a91d0 WatchSource:0}: Error finding container c0165edc9b112d0cedb920020a46938ee3cbb3ee35e833af8838871c7a1a91d0: Status 404 returned error can't find the container with id c0165edc9b112d0cedb920020a46938ee3cbb3ee35e833af8838871c7a1a91d0 Sep 08 14:22:13.955425 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.955387 2578 generic.go:356] "Generic (PLEG): container finished" podID="3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050" containerID="42e27fe7768233c026396528a2afa07ad42fd221ee18e1099b8781a94dc461a3" exitCode=0 Sep 08 14:22:13.955860 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.955470 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" event={"ID":"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050","Type":"ContainerDied","Data":"42e27fe7768233c026396528a2afa07ad42fd221ee18e1099b8781a94dc461a3"} Sep 08 14:22:13.956768 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:13.956739 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" event={"ID":"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9","Type":"ContainerStarted","Data":"c0165edc9b112d0cedb920020a46938ee3cbb3ee35e833af8838871c7a1a91d0"} Sep 08 14:22:16.544449 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.544417 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" Sep 08 14:22:16.651988 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.651962 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-bundle\") pod \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\" (UID: \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\") " Sep 08 14:22:16.652086 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.652007 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsf6c\" (UniqueName: \"kubernetes.io/projected/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-kube-api-access-vsf6c\") pod \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\" (UID: \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\") " Sep 08 14:22:16.652086 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.652066 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-util\") pod \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\" (UID: \"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050\") " Sep 08 14:22:16.652756 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.652693 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-bundle" (OuterVolumeSpecName: "bundle") pod "3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050" (UID: "3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:22:16.654044 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.654022 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-kube-api-access-vsf6c" (OuterVolumeSpecName: "kube-api-access-vsf6c") pod "3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050" (UID: "3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050"). InnerVolumeSpecName "kube-api-access-vsf6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:22:16.658505 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.658475 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-util" (OuterVolumeSpecName: "util") pod "3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050" (UID: "3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:22:16.752905 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.752872 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:22:16.752905 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.752898 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsf6c\" (UniqueName: \"kubernetes.io/projected/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-kube-api-access-vsf6c\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:22:16.752905 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.752908 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050-util\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:22:16.974446 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.974413 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" event={"ID":"3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050","Type":"ContainerDied","Data":"62e1a9d938df78947292be71ec90c91efc9d864d892359df770922addd152e69"} Sep 08 14:22:16.974599 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.974460 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62e1a9d938df78947292be71ec90c91efc9d864d892359df770922addd152e69" Sep 08 14:22:16.974599 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.974419 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gj58q" Sep 08 14:22:16.976273 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:16.976244 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" event={"ID":"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9","Type":"ContainerStarted","Data":"14f76d0f99a04e36c7983e4eb99eced608a349dc4e3aebe494e4999683f7ee3e"} Sep 08 14:22:17.982269 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:17.982233 2578 generic.go:356] "Generic (PLEG): container finished" podID="7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9" containerID="14f76d0f99a04e36c7983e4eb99eced608a349dc4e3aebe494e4999683f7ee3e" exitCode=0 Sep 08 14:22:17.982723 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:17.982294 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" event={"ID":"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9","Type":"ContainerDied","Data":"14f76d0f99a04e36c7983e4eb99eced608a349dc4e3aebe494e4999683f7ee3e"} Sep 08 14:22:18.988276 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:18.988237 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" event={"ID":"7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9","Type":"ContainerStarted","Data":"097aec07f073645e1534b9d3d6ac1d4b495a4c4412f6ca3cc38e8fa528944bf8"} Sep 08 14:22:18.988707 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:18.988363 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:19.013364 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:19.013318 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" podStartSLOduration=2.43733698 podStartE2EDuration="7.013303359s" totalImagesPullingTime="3.55369059s" totalInitContainerRuntime="1.022275789s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:22:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:22:19.011540766 +0000 UTC m=+351.295857882" watchObservedRunningTime="2026-09-08 14:22:19.013303359 +0000 UTC m=+351.297620474" Sep 08 14:22:29.995648 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:29.995617 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-566d57d45b-gwwdv" Sep 08 14:22:34.684676 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.684643 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb"] Sep 08 14:22:34.685052 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.684964 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050" containerName="util" Sep 08 14:22:34.685052 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.684976 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050" containerName="util" Sep 08 14:22:34.685052 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.684993 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050" containerName="extract" Sep 08 14:22:34.685052 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.684998 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050" containerName="extract" Sep 08 14:22:34.685052 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.685011 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050" containerName="pull" Sep 08 14:22:34.685052 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.685016 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050" containerName="pull" Sep 08 14:22:34.685234 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.685076 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3880ac1a-3eb6-4f74-b1d1-cbe6d5aab050" containerName="extract" Sep 08 14:22:34.693962 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.693941 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.696307 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.696284 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 14:22:34.696451 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.696332 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5p5vb\"" Sep 08 14:22:34.696451 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.696290 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 14:22:34.696451 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.696288 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 08 14:22:34.696451 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.696396 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 08 14:22:34.696451 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.696419 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 08 14:22:34.696694 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.696676 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 14:22:34.701897 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.701623 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb"] Sep 08 14:22:34.786593 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.786561 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.786799 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.786597 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.786799 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.786686 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50071dc0-0450-479d-9114-9a86d53d27d8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.786799 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.786718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50071dc0-0450-479d-9114-9a86d53d27d8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.786799 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.786787 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.786954 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.786818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50071dc0-0450-479d-9114-9a86d53d27d8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.786954 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.786849 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qswgs\" (UniqueName: \"kubernetes.io/projected/50071dc0-0450-479d-9114-9a86d53d27d8-kube-api-access-qswgs\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.887741 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.887694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50071dc0-0450-479d-9114-9a86d53d27d8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.887923 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.887765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50071dc0-0450-479d-9114-9a86d53d27d8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.887923 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.887807 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.887923 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.887846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50071dc0-0450-479d-9114-9a86d53d27d8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.887923 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.887882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qswgs\" (UniqueName: \"kubernetes.io/projected/50071dc0-0450-479d-9114-9a86d53d27d8-kube-api-access-qswgs\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.888134 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.887945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.888134 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.887973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.888660 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.888608 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50071dc0-0450-479d-9114-9a86d53d27d8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.890321 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.890299 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.890452 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.890324 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50071dc0-0450-479d-9114-9a86d53d27d8-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.890452 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.890387 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.890625 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.890477 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.897516 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.897491 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qswgs\" (UniqueName: \"kubernetes.io/projected/50071dc0-0450-479d-9114-9a86d53d27d8-kube-api-access-qswgs\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:34.898835 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:34.898816 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50071dc0-0450-479d-9114-9a86d53d27d8-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-5mhhb\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:35.004688 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:35.004659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:35.277948 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:35.277920 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb"] Sep 08 14:22:35.279751 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:22:35.279700 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50071dc0_0450_479d_9114_9a86d53d27d8.slice/crio-c37b94332ab47868d64a72980150e665544aeaf99aa0eefe8094fd95d0001645 WatchSource:0}: Error finding container c37b94332ab47868d64a72980150e665544aeaf99aa0eefe8094fd95d0001645: Status 404 returned error can't find the container with id c37b94332ab47868d64a72980150e665544aeaf99aa0eefe8094fd95d0001645 Sep 08 14:22:36.065783 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:36.065745 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" event={"ID":"50071dc0-0450-479d-9114-9a86d53d27d8","Type":"ContainerStarted","Data":"c37b94332ab47868d64a72980150e665544aeaf99aa0eefe8094fd95d0001645"} Sep 08 14:22:37.959042 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:37.959006 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 08 14:22:37.959307 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:37.959073 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 08 14:22:38.078868 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:38.078834 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" event={"ID":"50071dc0-0450-479d-9114-9a86d53d27d8","Type":"ContainerStarted","Data":"bc83f0cd437501df111c2fdc409ed1774befa0dabaa6d9b98f6e44315c6780de"} Sep 08 14:22:38.079054 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:38.078970 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:38.100216 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:38.100160 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" podStartSLOduration=1.4234663379999999 podStartE2EDuration="4.100139575s" totalImagesPullingTime="2.676673237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:22:38.097612035 +0000 UTC m=+370.381929152" watchObservedRunningTime="2026-09-08 14:22:38.100139575 +0000 UTC m=+370.384456690" Sep 08 14:22:39.087143 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:39.087110 2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-5mhhb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 08 14:22:39.087481 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:39.087169 2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" podUID="50071dc0-0450-479d-9114-9a86d53d27d8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 08 14:22:42.086963 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:42.086930 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:51.622298 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.622258 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f"] Sep 08 14:22:51.625631 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.625612 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:51.629030 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.629008 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 08 14:22:51.629138 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.629058 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 08 14:22:51.629138 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.629081 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 08 14:22:51.629138 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.629058 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wkspg\"" Sep 08 14:22:51.636649 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.636626 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f"] Sep 08 14:22:51.739756 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.739703 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09afb555-557d-44da-b8f1-0d387e2b5706-manager-config\") pod \"lws-controller-manager-655586cd5f-sjq9f\" (UID: \"09afb555-557d-44da-b8f1-0d387e2b5706\") " pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:51.739921 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.739836 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzr7x\" (UniqueName: \"kubernetes.io/projected/09afb555-557d-44da-b8f1-0d387e2b5706-kube-api-access-rzr7x\") pod \"lws-controller-manager-655586cd5f-sjq9f\" (UID: \"09afb555-557d-44da-b8f1-0d387e2b5706\") " pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:51.739921 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.739865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/09afb555-557d-44da-b8f1-0d387e2b5706-metrics-cert\") pod \"lws-controller-manager-655586cd5f-sjq9f\" (UID: \"09afb555-557d-44da-b8f1-0d387e2b5706\") " pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:51.739921 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.739883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09afb555-557d-44da-b8f1-0d387e2b5706-cert\") pod \"lws-controller-manager-655586cd5f-sjq9f\" (UID: \"09afb555-557d-44da-b8f1-0d387e2b5706\") " pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:51.840596 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.840557 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzr7x\" (UniqueName: \"kubernetes.io/projected/09afb555-557d-44da-b8f1-0d387e2b5706-kube-api-access-rzr7x\") pod \"lws-controller-manager-655586cd5f-sjq9f\" (UID: \"09afb555-557d-44da-b8f1-0d387e2b5706\") " pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:51.840596 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.840600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/09afb555-557d-44da-b8f1-0d387e2b5706-metrics-cert\") pod \"lws-controller-manager-655586cd5f-sjq9f\" (UID: \"09afb555-557d-44da-b8f1-0d387e2b5706\") " pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:51.840838 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.840619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09afb555-557d-44da-b8f1-0d387e2b5706-cert\") pod \"lws-controller-manager-655586cd5f-sjq9f\" (UID: \"09afb555-557d-44da-b8f1-0d387e2b5706\") " pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:51.840838 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.840658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09afb555-557d-44da-b8f1-0d387e2b5706-manager-config\") pod \"lws-controller-manager-655586cd5f-sjq9f\" (UID: \"09afb555-557d-44da-b8f1-0d387e2b5706\") " pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:51.841392 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.841285 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/09afb555-557d-44da-b8f1-0d387e2b5706-manager-config\") pod \"lws-controller-manager-655586cd5f-sjq9f\" (UID: \"09afb555-557d-44da-b8f1-0d387e2b5706\") " pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:51.843206 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.843183 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09afb555-557d-44da-b8f1-0d387e2b5706-cert\") pod \"lws-controller-manager-655586cd5f-sjq9f\" (UID: \"09afb555-557d-44da-b8f1-0d387e2b5706\") " pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:51.843314 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.843221 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/09afb555-557d-44da-b8f1-0d387e2b5706-metrics-cert\") pod \"lws-controller-manager-655586cd5f-sjq9f\" (UID: \"09afb555-557d-44da-b8f1-0d387e2b5706\") " pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:51.849583 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.849564 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzr7x\" (UniqueName: \"kubernetes.io/projected/09afb555-557d-44da-b8f1-0d387e2b5706-kube-api-access-rzr7x\") pod \"lws-controller-manager-655586cd5f-sjq9f\" (UID: \"09afb555-557d-44da-b8f1-0d387e2b5706\") " pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:51.934807 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:51.934703 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:52.215159 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:52.215135 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f"] Sep 08 14:22:52.217014 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:22:52.216988 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09afb555_557d_44da_b8f1_0d387e2b5706.slice/crio-22800086ad46dad9de717ef2a38d2852ee80491ad4ae96c196dca767fdef5324 WatchSource:0}: Error finding container 22800086ad46dad9de717ef2a38d2852ee80491ad4ae96c196dca767fdef5324: Status 404 returned error can't find the container with id 22800086ad46dad9de717ef2a38d2852ee80491ad4ae96c196dca767fdef5324 Sep 08 14:22:53.157955 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:53.157923 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" event={"ID":"09afb555-557d-44da-b8f1-0d387e2b5706","Type":"ContainerStarted","Data":"22800086ad46dad9de717ef2a38d2852ee80491ad4ae96c196dca767fdef5324"} Sep 08 14:22:55.173284 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:55.173242 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" event={"ID":"09afb555-557d-44da-b8f1-0d387e2b5706","Type":"ContainerStarted","Data":"c5edd7c2093e81368fe7d0e221a3947f4b167d86ecdd88b4dc2c5d01c7c6e6ff"} Sep 08 14:22:55.173646 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:55.173340 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:22:55.192552 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:55.192501 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" podStartSLOduration=2.2719735930000002 podStartE2EDuration="4.192486447s" totalImagesPullingTime="1.920512854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:22:55.191337882 +0000 UTC m=+387.475654997" watchObservedRunningTime="2026-09-08 14:22:55.192486447 +0000 UTC m=+387.476803561" Sep 08 14:22:56.759356 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:56.759321 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb"] Sep 08 14:22:56.759786 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:56.759528 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" podUID="50071dc0-0450-479d-9114-9a86d53d27d8" containerName="discovery" containerID="cri-o://bc83f0cd437501df111c2fdc409ed1774befa0dabaa6d9b98f6e44315c6780de" gracePeriod=30 Sep 08 14:22:57.005349 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.005325 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:57.088771 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.088689 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-istio-kubeconfig\") pod \"50071dc0-0450-479d-9114-9a86d53d27d8\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " Sep 08 14:22:57.088771 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.088745 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50071dc0-0450-479d-9114-9a86d53d27d8-istio-token\") pod \"50071dc0-0450-479d-9114-9a86d53d27d8\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " Sep 08 14:22:57.088987 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.088796 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-cacerts\") pod \"50071dc0-0450-479d-9114-9a86d53d27d8\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " Sep 08 14:22:57.088987 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.088820 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50071dc0-0450-479d-9114-9a86d53d27d8-istio-csr-ca-configmap\") pod \"50071dc0-0450-479d-9114-9a86d53d27d8\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " Sep 08 14:22:57.088987 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.088846 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-istio-csr-dns-cert\") pod \"50071dc0-0450-479d-9114-9a86d53d27d8\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " Sep 08 14:22:57.088987 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.088888 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qswgs\" (UniqueName: \"kubernetes.io/projected/50071dc0-0450-479d-9114-9a86d53d27d8-kube-api-access-qswgs\") pod \"50071dc0-0450-479d-9114-9a86d53d27d8\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " Sep 08 14:22:57.088987 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.088944 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50071dc0-0450-479d-9114-9a86d53d27d8-local-certs\") pod \"50071dc0-0450-479d-9114-9a86d53d27d8\" (UID: \"50071dc0-0450-479d-9114-9a86d53d27d8\") " Sep 08 14:22:57.089239 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.089203 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50071dc0-0450-479d-9114-9a86d53d27d8-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "50071dc0-0450-479d-9114-9a86d53d27d8" (UID: "50071dc0-0450-479d-9114-9a86d53d27d8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:22:57.091298 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.091272 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-cacerts" (OuterVolumeSpecName: "cacerts") pod "50071dc0-0450-479d-9114-9a86d53d27d8" (UID: "50071dc0-0450-479d-9114-9a86d53d27d8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:22:57.091419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.091371 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50071dc0-0450-479d-9114-9a86d53d27d8-istio-token" (OuterVolumeSpecName: "istio-token") pod "50071dc0-0450-479d-9114-9a86d53d27d8" (UID: "50071dc0-0450-479d-9114-9a86d53d27d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:22:57.091544 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.091521 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "50071dc0-0450-479d-9114-9a86d53d27d8" (UID: "50071dc0-0450-479d-9114-9a86d53d27d8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:22:57.091904 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.091876 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50071dc0-0450-479d-9114-9a86d53d27d8-kube-api-access-qswgs" (OuterVolumeSpecName: "kube-api-access-qswgs") pod "50071dc0-0450-479d-9114-9a86d53d27d8" (UID: "50071dc0-0450-479d-9114-9a86d53d27d8"). InnerVolumeSpecName "kube-api-access-qswgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:22:57.091904 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.091883 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50071dc0-0450-479d-9114-9a86d53d27d8-local-certs" (OuterVolumeSpecName: "local-certs") pod "50071dc0-0450-479d-9114-9a86d53d27d8" (UID: "50071dc0-0450-479d-9114-9a86d53d27d8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:22:57.091904 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.091893 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "50071dc0-0450-479d-9114-9a86d53d27d8" (UID: "50071dc0-0450-479d-9114-9a86d53d27d8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:22:57.183649 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.183615 2578 generic.go:356] "Generic (PLEG): container finished" podID="50071dc0-0450-479d-9114-9a86d53d27d8" containerID="bc83f0cd437501df111c2fdc409ed1774befa0dabaa6d9b98f6e44315c6780de" exitCode=0 Sep 08 14:22:57.183803 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.183656 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" Sep 08 14:22:57.183803 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.183697 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" event={"ID":"50071dc0-0450-479d-9114-9a86d53d27d8","Type":"ContainerDied","Data":"bc83f0cd437501df111c2fdc409ed1774befa0dabaa6d9b98f6e44315c6780de"} Sep 08 14:22:57.183803 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.183790 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb" event={"ID":"50071dc0-0450-479d-9114-9a86d53d27d8","Type":"ContainerDied","Data":"c37b94332ab47868d64a72980150e665544aeaf99aa0eefe8094fd95d0001645"} Sep 08 14:22:57.183951 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.183813 2578 scope.go:117] "RemoveContainer" containerID="bc83f0cd437501df111c2fdc409ed1774befa0dabaa6d9b98f6e44315c6780de" Sep 08 14:22:57.189866 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.189841 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qswgs\" (UniqueName: \"kubernetes.io/projected/50071dc0-0450-479d-9114-9a86d53d27d8-kube-api-access-qswgs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:22:57.189970 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.189868 2578 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/50071dc0-0450-479d-9114-9a86d53d27d8-local-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:22:57.189970 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.189884 2578 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-istio-kubeconfig\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:22:57.189970 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.189897 2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50071dc0-0450-479d-9114-9a86d53d27d8-istio-token\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:22:57.189970 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.189912 2578 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-cacerts\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:22:57.189970 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.189925 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/50071dc0-0450-479d-9114-9a86d53d27d8-istio-csr-ca-configmap\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:22:57.189970 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.189938 2578 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/50071dc0-0450-479d-9114-9a86d53d27d8-istio-csr-dns-cert\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:22:57.192524 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.192508 2578 scope.go:117] "RemoveContainer" containerID="bc83f0cd437501df111c2fdc409ed1774befa0dabaa6d9b98f6e44315c6780de" Sep 08 14:22:57.192804 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:22:57.192777 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc83f0cd437501df111c2fdc409ed1774befa0dabaa6d9b98f6e44315c6780de\": container with ID starting with bc83f0cd437501df111c2fdc409ed1774befa0dabaa6d9b98f6e44315c6780de not found: ID does not exist" containerID="bc83f0cd437501df111c2fdc409ed1774befa0dabaa6d9b98f6e44315c6780de" Sep 08 14:22:57.192847 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.192814 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc83f0cd437501df111c2fdc409ed1774befa0dabaa6d9b98f6e44315c6780de"} err="failed to get container status \"bc83f0cd437501df111c2fdc409ed1774befa0dabaa6d9b98f6e44315c6780de\": rpc error: code = NotFound desc = could not find container \"bc83f0cd437501df111c2fdc409ed1774befa0dabaa6d9b98f6e44315c6780de\": container with ID starting with bc83f0cd437501df111c2fdc409ed1774befa0dabaa6d9b98f6e44315c6780de not found: ID does not exist" Sep 08 14:22:57.208217 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.208182 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb"] Sep 08 14:22:57.209952 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.209931 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-5mhhb"] Sep 08 14:22:57.973210 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:57.973181 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dbfdccd47-kwgls"] Sep 08 14:22:58.241344 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:22:58.241269 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50071dc0-0450-479d-9114-9a86d53d27d8" path="/var/lib/kubelet/pods/50071dc0-0450-479d-9114-9a86d53d27d8/volumes" Sep 08 14:23:06.180081 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:06.180048 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-655586cd5f-sjq9f" Sep 08 14:23:22.993232 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:22.993177 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7dbfdccd47-kwgls" podUID="2b4b13d3-2198-4062-9486-d31df6aa50a8" containerName="console" containerID="cri-o://1c4cdcf9cf99946150529cb68dc0fa4424393df564bd477f8c4278e73b9faba6" gracePeriod=15 Sep 08 14:23:23.233216 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.233188 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:23:23.310177 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.310103 2578 generic.go:356] "Generic (PLEG): container finished" podID="2b4b13d3-2198-4062-9486-d31df6aa50a8" containerID="1c4cdcf9cf99946150529cb68dc0fa4424393df564bd477f8c4278e73b9faba6" exitCode=0 Sep 08 14:23:23.310177 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.310155 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7dbfdccd47-kwgls" Sep 08 14:23:23.310407 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.310189 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbfdccd47-kwgls" event={"ID":"2b4b13d3-2198-4062-9486-d31df6aa50a8","Type":"ContainerDied","Data":"1c4cdcf9cf99946150529cb68dc0fa4424393df564bd477f8c4278e73b9faba6"} Sep 08 14:23:23.310407 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.310244 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7dbfdccd47-kwgls" event={"ID":"2b4b13d3-2198-4062-9486-d31df6aa50a8","Type":"ContainerDied","Data":"336f48cec137ee7921311e775cbaee5cdb22e45926ff2e610c988e275f688abd"} Sep 08 14:23:23.310407 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.310269 2578 scope.go:117] "RemoveContainer" containerID="1c4cdcf9cf99946150529cb68dc0fa4424393df564bd477f8c4278e73b9faba6" Sep 08 14:23:23.318055 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.318034 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzrrn\" (UniqueName: \"kubernetes.io/projected/2b4b13d3-2198-4062-9486-d31df6aa50a8-kube-api-access-nzrrn\") pod \"2b4b13d3-2198-4062-9486-d31df6aa50a8\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " Sep 08 14:23:23.318182 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.318070 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-service-ca\") pod \"2b4b13d3-2198-4062-9486-d31df6aa50a8\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " Sep 08 14:23:23.318182 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.318103 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-trusted-ca-bundle\") pod \"2b4b13d3-2198-4062-9486-d31df6aa50a8\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " Sep 08 14:23:23.318182 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.318126 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-serving-cert\") pod \"2b4b13d3-2198-4062-9486-d31df6aa50a8\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " Sep 08 14:23:23.318182 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.318150 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-config\") pod \"2b4b13d3-2198-4062-9486-d31df6aa50a8\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " Sep 08 14:23:23.318182 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.318177 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-oauth-serving-cert\") pod \"2b4b13d3-2198-4062-9486-d31df6aa50a8\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " Sep 08 14:23:23.318420 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.318239 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-oauth-config\") pod \"2b4b13d3-2198-4062-9486-d31df6aa50a8\" (UID: \"2b4b13d3-2198-4062-9486-d31df6aa50a8\") " Sep 08 14:23:23.318481 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.318415 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-service-ca" (OuterVolumeSpecName: "service-ca") pod "2b4b13d3-2198-4062-9486-d31df6aa50a8" (UID: "2b4b13d3-2198-4062-9486-d31df6aa50a8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:23:23.318538 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.318520 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:23.318742 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.318701 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2b4b13d3-2198-4062-9486-d31df6aa50a8" (UID: "2b4b13d3-2198-4062-9486-d31df6aa50a8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:23:23.318843 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.318766 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-config" (OuterVolumeSpecName: "console-config") pod "2b4b13d3-2198-4062-9486-d31df6aa50a8" (UID: "2b4b13d3-2198-4062-9486-d31df6aa50a8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:23:23.318889 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.318838 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2b4b13d3-2198-4062-9486-d31df6aa50a8" (UID: "2b4b13d3-2198-4062-9486-d31df6aa50a8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:23:23.319304 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.319281 2578 scope.go:117] "RemoveContainer" containerID="1c4cdcf9cf99946150529cb68dc0fa4424393df564bd477f8c4278e73b9faba6" Sep 08 14:23:23.319596 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:23:23.319577 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c4cdcf9cf99946150529cb68dc0fa4424393df564bd477f8c4278e73b9faba6\": container with ID starting with 1c4cdcf9cf99946150529cb68dc0fa4424393df564bd477f8c4278e73b9faba6 not found: ID does not exist" containerID="1c4cdcf9cf99946150529cb68dc0fa4424393df564bd477f8c4278e73b9faba6" Sep 08 14:23:23.319686 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.319601 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c4cdcf9cf99946150529cb68dc0fa4424393df564bd477f8c4278e73b9faba6"} err="failed to get container status \"1c4cdcf9cf99946150529cb68dc0fa4424393df564bd477f8c4278e73b9faba6\": rpc error: code = NotFound desc = could not find container \"1c4cdcf9cf99946150529cb68dc0fa4424393df564bd477f8c4278e73b9faba6\": container with ID starting with 1c4cdcf9cf99946150529cb68dc0fa4424393df564bd477f8c4278e73b9faba6 not found: ID does not exist" Sep 08 14:23:23.320392 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.320369 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b4b13d3-2198-4062-9486-d31df6aa50a8-kube-api-access-nzrrn" (OuterVolumeSpecName: "kube-api-access-nzrrn") pod "2b4b13d3-2198-4062-9486-d31df6aa50a8" (UID: "2b4b13d3-2198-4062-9486-d31df6aa50a8"). InnerVolumeSpecName "kube-api-access-nzrrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:23:23.320556 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.320536 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2b4b13d3-2198-4062-9486-d31df6aa50a8" (UID: "2b4b13d3-2198-4062-9486-d31df6aa50a8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:23:23.320819 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.320793 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2b4b13d3-2198-4062-9486-d31df6aa50a8" (UID: "2b4b13d3-2198-4062-9486-d31df6aa50a8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:23:23.419419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.419381 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-oauth-config\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:23.419419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.419414 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzrrn\" (UniqueName: \"kubernetes.io/projected/2b4b13d3-2198-4062-9486-d31df6aa50a8-kube-api-access-nzrrn\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:23.419419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.419425 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-trusted-ca-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:23.419419 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.419433 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-serving-cert\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:23.419641 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.419442 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-console-config\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:23.419641 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.419450 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b4b13d3-2198-4062-9486-d31df6aa50a8-oauth-serving-cert\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:23.632367 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.632336 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7dbfdccd47-kwgls"] Sep 08 14:23:23.636108 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:23.636086 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7dbfdccd47-kwgls"] Sep 08 14:23:24.241395 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:24.241362 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b4b13d3-2198-4062-9486-d31df6aa50a8" path="/var/lib/kubelet/pods/2b4b13d3-2198-4062-9486-d31df6aa50a8/volumes" Sep 08 14:23:48.642146 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.642068 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9"] Sep 08 14:23:48.642552 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.642380 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b4b13d3-2198-4062-9486-d31df6aa50a8" containerName="console" Sep 08 14:23:48.642552 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.642391 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b4b13d3-2198-4062-9486-d31df6aa50a8" containerName="console" Sep 08 14:23:48.642552 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.642400 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50071dc0-0450-479d-9114-9a86d53d27d8" containerName="discovery" Sep 08 14:23:48.642552 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.642406 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="50071dc0-0450-479d-9114-9a86d53d27d8" containerName="discovery" Sep 08 14:23:48.642552 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.642462 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b4b13d3-2198-4062-9486-d31df6aa50a8" containerName="console" Sep 08 14:23:48.642552 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.642470 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="50071dc0-0450-479d-9114-9a86d53d27d8" containerName="discovery" Sep 08 14:23:48.647021 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.647005 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" Sep 08 14:23:48.649228 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.649211 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 08 14:23:48.649339 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.649242 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-snnvk\"" Sep 08 14:23:48.650079 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.650060 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 08 14:23:48.654980 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.654959 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9"] Sep 08 14:23:48.729538 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.729506 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd"] Sep 08 14:23:48.733002 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.732987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" Sep 08 14:23:48.741411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.741386 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd"] Sep 08 14:23:48.745108 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.745086 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6l88\" (UniqueName: \"kubernetes.io/projected/80a07a1b-8956-4f86-8c82-92a0cbc47420-kube-api-access-p6l88\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9\" (UID: \"80a07a1b-8956-4f86-8c82-92a0cbc47420\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" Sep 08 14:23:48.745206 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.745127 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80a07a1b-8956-4f86-8c82-92a0cbc47420-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9\" (UID: \"80a07a1b-8956-4f86-8c82-92a0cbc47420\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" Sep 08 14:23:48.745250 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.745206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80a07a1b-8956-4f86-8c82-92a0cbc47420-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9\" (UID: \"80a07a1b-8956-4f86-8c82-92a0cbc47420\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" Sep 08 14:23:48.830528 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.830484 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp"] Sep 08 14:23:48.834045 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.834030 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" Sep 08 14:23:48.842164 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.842143 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp"] Sep 08 14:23:48.845755 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.845707 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80a07a1b-8956-4f86-8c82-92a0cbc47420-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9\" (UID: \"80a07a1b-8956-4f86-8c82-92a0cbc47420\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" Sep 08 14:23:48.845843 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.845826 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5kgc\" (UniqueName: \"kubernetes.io/projected/171a4baa-eb9a-4048-a9f2-c8561943bedf-kube-api-access-s5kgc\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd\" (UID: \"171a4baa-eb9a-4048-a9f2-c8561943bedf\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" Sep 08 14:23:48.845888 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.845865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80a07a1b-8956-4f86-8c82-92a0cbc47420-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9\" (UID: \"80a07a1b-8956-4f86-8c82-92a0cbc47420\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" Sep 08 14:23:48.845959 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.845942 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/171a4baa-eb9a-4048-a9f2-c8561943bedf-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd\" (UID: \"171a4baa-eb9a-4048-a9f2-c8561943bedf\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" Sep 08 14:23:48.846020 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.845994 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6l88\" (UniqueName: \"kubernetes.io/projected/80a07a1b-8956-4f86-8c82-92a0cbc47420-kube-api-access-p6l88\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9\" (UID: \"80a07a1b-8956-4f86-8c82-92a0cbc47420\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" Sep 08 14:23:48.846077 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.846034 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80a07a1b-8956-4f86-8c82-92a0cbc47420-bundle\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9\" (UID: \"80a07a1b-8956-4f86-8c82-92a0cbc47420\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" Sep 08 14:23:48.846077 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.846037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/171a4baa-eb9a-4048-a9f2-c8561943bedf-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd\" (UID: \"171a4baa-eb9a-4048-a9f2-c8561943bedf\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" Sep 08 14:23:48.846194 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.846170 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80a07a1b-8956-4f86-8c82-92a0cbc47420-util\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9\" (UID: \"80a07a1b-8956-4f86-8c82-92a0cbc47420\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" Sep 08 14:23:48.855425 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.855405 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6l88\" (UniqueName: \"kubernetes.io/projected/80a07a1b-8956-4f86-8c82-92a0cbc47420-kube-api-access-p6l88\") pod \"71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9\" (UID: \"80a07a1b-8956-4f86-8c82-92a0cbc47420\") " pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" Sep 08 14:23:48.930965 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.930888 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z"] Sep 08 14:23:48.934553 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.934537 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" Sep 08 14:23:48.943590 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.943569 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z"] Sep 08 14:23:48.947148 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.947129 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf9vz\" (UniqueName: \"kubernetes.io/projected/93aa8f77-82c3-4942-b0b7-fb60db1592fb-kube-api-access-rf9vz\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp\" (UID: \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" Sep 08 14:23:48.947281 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.947162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/171a4baa-eb9a-4048-a9f2-c8561943bedf-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd\" (UID: \"171a4baa-eb9a-4048-a9f2-c8561943bedf\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" Sep 08 14:23:48.947281 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.947193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/171a4baa-eb9a-4048-a9f2-c8561943bedf-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd\" (UID: \"171a4baa-eb9a-4048-a9f2-c8561943bedf\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" Sep 08 14:23:48.947281 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.947244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93aa8f77-82c3-4942-b0b7-fb60db1592fb-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp\" (UID: \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" Sep 08 14:23:48.947438 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.947291 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93aa8f77-82c3-4942-b0b7-fb60db1592fb-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp\" (UID: \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" Sep 08 14:23:48.947438 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.947413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5kgc\" (UniqueName: \"kubernetes.io/projected/171a4baa-eb9a-4048-a9f2-c8561943bedf-kube-api-access-s5kgc\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd\" (UID: \"171a4baa-eb9a-4048-a9f2-c8561943bedf\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" Sep 08 14:23:48.947521 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.947510 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/171a4baa-eb9a-4048-a9f2-c8561943bedf-bundle\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd\" (UID: \"171a4baa-eb9a-4048-a9f2-c8561943bedf\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" Sep 08 14:23:48.947567 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.947547 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/171a4baa-eb9a-4048-a9f2-c8561943bedf-util\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd\" (UID: \"171a4baa-eb9a-4048-a9f2-c8561943bedf\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" Sep 08 14:23:48.956022 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.956001 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5kgc\" (UniqueName: \"kubernetes.io/projected/171a4baa-eb9a-4048-a9f2-c8561943bedf-kube-api-access-s5kgc\") pod \"eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd\" (UID: \"171a4baa-eb9a-4048-a9f2-c8561943bedf\") " pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" Sep 08 14:23:48.956772 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:48.956751 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" Sep 08 14:23:49.042224 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.042198 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" Sep 08 14:23:49.048554 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.048532 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93aa8f77-82c3-4942-b0b7-fb60db1592fb-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp\" (UID: \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" Sep 08 14:23:49.048631 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.048563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93aa8f77-82c3-4942-b0b7-fb60db1592fb-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp\" (UID: \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" Sep 08 14:23:49.048631 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.048584 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtr76\" (UniqueName: \"kubernetes.io/projected/ca2453a3-e95b-4d23-88df-22fc028397c4-kube-api-access-dtr76\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z\" (UID: \"ca2453a3-e95b-4d23-88df-22fc028397c4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" Sep 08 14:23:49.048748 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.048653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca2453a3-e95b-4d23-88df-22fc028397c4-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z\" (UID: \"ca2453a3-e95b-4d23-88df-22fc028397c4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" Sep 08 14:23:49.048748 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.048673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca2453a3-e95b-4d23-88df-22fc028397c4-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z\" (UID: \"ca2453a3-e95b-4d23-88df-22fc028397c4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" Sep 08 14:23:49.048748 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.048697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf9vz\" (UniqueName: \"kubernetes.io/projected/93aa8f77-82c3-4942-b0b7-fb60db1592fb-kube-api-access-rf9vz\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp\" (UID: \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" Sep 08 14:23:49.048964 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.048947 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93aa8f77-82c3-4942-b0b7-fb60db1592fb-util\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp\" (UID: \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" Sep 08 14:23:49.049037 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.049004 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93aa8f77-82c3-4942-b0b7-fb60db1592fb-bundle\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp\" (UID: \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" Sep 08 14:23:49.057633 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.057609 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf9vz\" (UniqueName: \"kubernetes.io/projected/93aa8f77-82c3-4942-b0b7-fb60db1592fb-kube-api-access-rf9vz\") pod \"cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp\" (UID: \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\") " pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" Sep 08 14:23:49.144015 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.143985 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" Sep 08 14:23:49.149349 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.149326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca2453a3-e95b-4d23-88df-22fc028397c4-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z\" (UID: \"ca2453a3-e95b-4d23-88df-22fc028397c4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" Sep 08 14:23:49.149421 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.149375 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca2453a3-e95b-4d23-88df-22fc028397c4-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z\" (UID: \"ca2453a3-e95b-4d23-88df-22fc028397c4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" Sep 08 14:23:49.149479 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.149446 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtr76\" (UniqueName: \"kubernetes.io/projected/ca2453a3-e95b-4d23-88df-22fc028397c4-kube-api-access-dtr76\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z\" (UID: \"ca2453a3-e95b-4d23-88df-22fc028397c4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" Sep 08 14:23:49.149755 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.149714 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca2453a3-e95b-4d23-88df-22fc028397c4-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z\" (UID: \"ca2453a3-e95b-4d23-88df-22fc028397c4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" Sep 08 14:23:49.149821 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.149755 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca2453a3-e95b-4d23-88df-22fc028397c4-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z\" (UID: \"ca2453a3-e95b-4d23-88df-22fc028397c4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" Sep 08 14:23:49.158195 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.158173 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtr76\" (UniqueName: \"kubernetes.io/projected/ca2453a3-e95b-4d23-88df-22fc028397c4-kube-api-access-dtr76\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z\" (UID: \"ca2453a3-e95b-4d23-88df-22fc028397c4\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" Sep 08 14:23:49.232126 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.232099 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9"] Sep 08 14:23:49.244434 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.244414 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" Sep 08 14:23:49.317908 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.317883 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd"] Sep 08 14:23:49.319083 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:23:49.319056 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod171a4baa_eb9a_4048_a9f2_c8561943bedf.slice/crio-97f6a201ddbfdf139b165e7a467df4c43182e3b1490dbfb0a9d649cfb166fbf6 WatchSource:0}: Error finding container 97f6a201ddbfdf139b165e7a467df4c43182e3b1490dbfb0a9d649cfb166fbf6: Status 404 returned error can't find the container with id 97f6a201ddbfdf139b165e7a467df4c43182e3b1490dbfb0a9d649cfb166fbf6 Sep 08 14:23:49.417389 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.417357 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp"] Sep 08 14:23:49.431972 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.431946 2578 generic.go:356] "Generic (PLEG): container finished" podID="80a07a1b-8956-4f86-8c82-92a0cbc47420" containerID="241d7fc06f17addb990105b147140b7952cc5066d4c19e827e51edd477344c3f" exitCode=0 Sep 08 14:23:49.432093 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.432021 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" event={"ID":"80a07a1b-8956-4f86-8c82-92a0cbc47420","Type":"ContainerDied","Data":"241d7fc06f17addb990105b147140b7952cc5066d4c19e827e51edd477344c3f"} Sep 08 14:23:49.432093 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.432074 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" event={"ID":"80a07a1b-8956-4f86-8c82-92a0cbc47420","Type":"ContainerStarted","Data":"7d4bc3b298017d22ebee06d891e14139b68432a2c79bcd5a781a50a26bad6b22"} Sep 08 14:23:49.433394 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.433370 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" event={"ID":"171a4baa-eb9a-4048-a9f2-c8561943bedf","Type":"ContainerStarted","Data":"97f6a201ddbfdf139b165e7a467df4c43182e3b1490dbfb0a9d649cfb166fbf6"} Sep 08 14:23:49.444051 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:23:49.444030 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93aa8f77_82c3_4942_b0b7_fb60db1592fb.slice/crio-aa9ed1884ad74025f992485f8c210229a8102d2ca15e7f9b447a4390c55c99c4 WatchSource:0}: Error finding container aa9ed1884ad74025f992485f8c210229a8102d2ca15e7f9b447a4390c55c99c4: Status 404 returned error can't find the container with id aa9ed1884ad74025f992485f8c210229a8102d2ca15e7f9b447a4390c55c99c4 Sep 08 14:23:49.519443 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:49.519420 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z"] Sep 08 14:23:49.614640 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:23:49.614611 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2453a3_e95b_4d23_88df_22fc028397c4.slice/crio-74cd8856f715e580008de2a346d28138fb8458f75a9dfd4fd8cedb6310877ac1 WatchSource:0}: Error finding container 74cd8856f715e580008de2a346d28138fb8458f75a9dfd4fd8cedb6310877ac1: Status 404 returned error can't find the container with id 74cd8856f715e580008de2a346d28138fb8458f75a9dfd4fd8cedb6310877ac1 Sep 08 14:23:50.439841 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:50.439803 2578 generic.go:356] "Generic (PLEG): container finished" podID="171a4baa-eb9a-4048-a9f2-c8561943bedf" containerID="208812362aae2dd8e4265ea9436b14df6ae3f0447256c2774cffc383e0c96294" exitCode=0 Sep 08 14:23:50.440259 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:50.439872 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" event={"ID":"171a4baa-eb9a-4048-a9f2-c8561943bedf","Type":"ContainerDied","Data":"208812362aae2dd8e4265ea9436b14df6ae3f0447256c2774cffc383e0c96294"} Sep 08 14:23:50.441512 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:50.441489 2578 generic.go:356] "Generic (PLEG): container finished" podID="ca2453a3-e95b-4d23-88df-22fc028397c4" containerID="5a08f0b566baa80da10a37e643decf00cbb8436baea12175cb9c2b38e559cd33" exitCode=0 Sep 08 14:23:50.441589 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:50.441568 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" event={"ID":"ca2453a3-e95b-4d23-88df-22fc028397c4","Type":"ContainerDied","Data":"5a08f0b566baa80da10a37e643decf00cbb8436baea12175cb9c2b38e559cd33"} Sep 08 14:23:50.441643 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:50.441610 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" event={"ID":"ca2453a3-e95b-4d23-88df-22fc028397c4","Type":"ContainerStarted","Data":"74cd8856f715e580008de2a346d28138fb8458f75a9dfd4fd8cedb6310877ac1"} Sep 08 14:23:50.443450 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:50.443425 2578 generic.go:356] "Generic (PLEG): container finished" podID="93aa8f77-82c3-4942-b0b7-fb60db1592fb" containerID="e01d7a393ed9d1884f8e7dc898b0efd57f833ba0461143205e4a8022d82e55d2" exitCode=0 Sep 08 14:23:50.443566 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:50.443453 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" event={"ID":"93aa8f77-82c3-4942-b0b7-fb60db1592fb","Type":"ContainerDied","Data":"e01d7a393ed9d1884f8e7dc898b0efd57f833ba0461143205e4a8022d82e55d2"} Sep 08 14:23:50.443566 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:50.443496 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" event={"ID":"93aa8f77-82c3-4942-b0b7-fb60db1592fb","Type":"ContainerStarted","Data":"aa9ed1884ad74025f992485f8c210229a8102d2ca15e7f9b447a4390c55c99c4"} Sep 08 14:23:52.455264 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:52.455232 2578 generic.go:356] "Generic (PLEG): container finished" podID="171a4baa-eb9a-4048-a9f2-c8561943bedf" containerID="9117f49891dfe78e369b11a5b330f181ffd03487f3ef4d10478c67669a6c6e07" exitCode=0 Sep 08 14:23:52.455685 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:52.455301 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" event={"ID":"171a4baa-eb9a-4048-a9f2-c8561943bedf","Type":"ContainerDied","Data":"9117f49891dfe78e369b11a5b330f181ffd03487f3ef4d10478c67669a6c6e07"} Sep 08 14:23:52.457263 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:52.457239 2578 generic.go:356] "Generic (PLEG): container finished" podID="ca2453a3-e95b-4d23-88df-22fc028397c4" containerID="8a41e9d0570ce8be440e6b595ede0070750c3b5c956ab697b8c198a39f304afd" exitCode=0 Sep 08 14:23:52.457354 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:52.457303 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" event={"ID":"ca2453a3-e95b-4d23-88df-22fc028397c4","Type":"ContainerDied","Data":"8a41e9d0570ce8be440e6b595ede0070750c3b5c956ab697b8c198a39f304afd"} Sep 08 14:23:52.459561 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:52.459539 2578 generic.go:356] "Generic (PLEG): container finished" podID="93aa8f77-82c3-4942-b0b7-fb60db1592fb" containerID="4ed9251cee7421384453112a2dd3b2aeb53fab1fe20229d0b32a4bd007d073d7" exitCode=0 Sep 08 14:23:52.459640 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:52.459590 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" event={"ID":"93aa8f77-82c3-4942-b0b7-fb60db1592fb","Type":"ContainerDied","Data":"4ed9251cee7421384453112a2dd3b2aeb53fab1fe20229d0b32a4bd007d073d7"} Sep 08 14:23:52.461612 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:52.461595 2578 generic.go:356] "Generic (PLEG): container finished" podID="80a07a1b-8956-4f86-8c82-92a0cbc47420" containerID="77f5eee0a524fae6daac432bf6b76c44ff5cf6b35cccfda04d64e0d53d4d98d4" exitCode=0 Sep 08 14:23:52.461688 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:52.461634 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" event={"ID":"80a07a1b-8956-4f86-8c82-92a0cbc47420","Type":"ContainerDied","Data":"77f5eee0a524fae6daac432bf6b76c44ff5cf6b35cccfda04d64e0d53d4d98d4"} Sep 08 14:23:53.468848 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:53.468814 2578 generic.go:356] "Generic (PLEG): container finished" podID="93aa8f77-82c3-4942-b0b7-fb60db1592fb" containerID="3cf46523b23034809ddc76743ca9b78622862df03cb40db358776c0dc2f66118" exitCode=0 Sep 08 14:23:53.469303 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:53.468901 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" event={"ID":"93aa8f77-82c3-4942-b0b7-fb60db1592fb","Type":"ContainerDied","Data":"3cf46523b23034809ddc76743ca9b78622862df03cb40db358776c0dc2f66118"} Sep 08 14:23:53.470795 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:53.470771 2578 generic.go:356] "Generic (PLEG): container finished" podID="80a07a1b-8956-4f86-8c82-92a0cbc47420" containerID="d42428a4a3ae2aaeded3ae8f6a994fac6f05529c24ad6665c99dd7c4bdc17dcd" exitCode=0 Sep 08 14:23:53.470904 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:53.470852 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" event={"ID":"80a07a1b-8956-4f86-8c82-92a0cbc47420","Type":"ContainerDied","Data":"d42428a4a3ae2aaeded3ae8f6a994fac6f05529c24ad6665c99dd7c4bdc17dcd"} Sep 08 14:23:53.472655 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:53.472635 2578 generic.go:356] "Generic (PLEG): container finished" podID="171a4baa-eb9a-4048-a9f2-c8561943bedf" containerID="5a67e7987d595d4917905ecb5ea712090066989543749ab3844a8a58d6ae41ea" exitCode=0 Sep 08 14:23:53.472779 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:53.472703 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" event={"ID":"171a4baa-eb9a-4048-a9f2-c8561943bedf","Type":"ContainerDied","Data":"5a67e7987d595d4917905ecb5ea712090066989543749ab3844a8a58d6ae41ea"} Sep 08 14:23:53.474743 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:53.474710 2578 generic.go:356] "Generic (PLEG): container finished" podID="ca2453a3-e95b-4d23-88df-22fc028397c4" containerID="eada6a8342ae4c1316c9989c22430225d5937dc05ff35b952ae9b3e3f458f7e3" exitCode=0 Sep 08 14:23:53.474823 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:53.474761 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" event={"ID":"ca2453a3-e95b-4d23-88df-22fc028397c4","Type":"ContainerDied","Data":"eada6a8342ae4c1316c9989c22430225d5937dc05ff35b952ae9b3e3f458f7e3"} Sep 08 14:23:54.643487 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.643467 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" Sep 08 14:23:54.646835 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.646815 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" Sep 08 14:23:54.708911 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.708886 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" Sep 08 14:23:54.711605 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.711587 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" Sep 08 14:23:54.806650 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.806578 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/171a4baa-eb9a-4048-a9f2-c8561943bedf-bundle\") pod \"171a4baa-eb9a-4048-a9f2-c8561943bedf\" (UID: \"171a4baa-eb9a-4048-a9f2-c8561943bedf\") " Sep 08 14:23:54.806650 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.806615 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80a07a1b-8956-4f86-8c82-92a0cbc47420-bundle\") pod \"80a07a1b-8956-4f86-8c82-92a0cbc47420\" (UID: \"80a07a1b-8956-4f86-8c82-92a0cbc47420\") " Sep 08 14:23:54.806650 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.806638 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93aa8f77-82c3-4942-b0b7-fb60db1592fb-util\") pod \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\" (UID: \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\") " Sep 08 14:23:54.806939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.806687 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf9vz\" (UniqueName: \"kubernetes.io/projected/93aa8f77-82c3-4942-b0b7-fb60db1592fb-kube-api-access-rf9vz\") pod \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\" (UID: \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\") " Sep 08 14:23:54.806939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.806708 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80a07a1b-8956-4f86-8c82-92a0cbc47420-util\") pod \"80a07a1b-8956-4f86-8c82-92a0cbc47420\" (UID: \"80a07a1b-8956-4f86-8c82-92a0cbc47420\") " Sep 08 14:23:54.806939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.806745 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtr76\" (UniqueName: \"kubernetes.io/projected/ca2453a3-e95b-4d23-88df-22fc028397c4-kube-api-access-dtr76\") pod \"ca2453a3-e95b-4d23-88df-22fc028397c4\" (UID: \"ca2453a3-e95b-4d23-88df-22fc028397c4\") " Sep 08 14:23:54.806939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.806770 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93aa8f77-82c3-4942-b0b7-fb60db1592fb-bundle\") pod \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\" (UID: \"93aa8f77-82c3-4942-b0b7-fb60db1592fb\") " Sep 08 14:23:54.806939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.806808 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca2453a3-e95b-4d23-88df-22fc028397c4-bundle\") pod \"ca2453a3-e95b-4d23-88df-22fc028397c4\" (UID: \"ca2453a3-e95b-4d23-88df-22fc028397c4\") " Sep 08 14:23:54.806939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.806840 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/171a4baa-eb9a-4048-a9f2-c8561943bedf-util\") pod \"171a4baa-eb9a-4048-a9f2-c8561943bedf\" (UID: \"171a4baa-eb9a-4048-a9f2-c8561943bedf\") " Sep 08 14:23:54.806939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.806877 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6l88\" (UniqueName: \"kubernetes.io/projected/80a07a1b-8956-4f86-8c82-92a0cbc47420-kube-api-access-p6l88\") pod \"80a07a1b-8956-4f86-8c82-92a0cbc47420\" (UID: \"80a07a1b-8956-4f86-8c82-92a0cbc47420\") " Sep 08 14:23:54.806939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.806899 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5kgc\" (UniqueName: \"kubernetes.io/projected/171a4baa-eb9a-4048-a9f2-c8561943bedf-kube-api-access-s5kgc\") pod \"171a4baa-eb9a-4048-a9f2-c8561943bedf\" (UID: \"171a4baa-eb9a-4048-a9f2-c8561943bedf\") " Sep 08 14:23:54.806939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.806927 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca2453a3-e95b-4d23-88df-22fc028397c4-util\") pod \"ca2453a3-e95b-4d23-88df-22fc028397c4\" (UID: \"ca2453a3-e95b-4d23-88df-22fc028397c4\") " Sep 08 14:23:54.807603 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.807389 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a07a1b-8956-4f86-8c82-92a0cbc47420-bundle" (OuterVolumeSpecName: "bundle") pod "80a07a1b-8956-4f86-8c82-92a0cbc47420" (UID: "80a07a1b-8956-4f86-8c82-92a0cbc47420"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:23:54.807693 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.807668 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171a4baa-eb9a-4048-a9f2-c8561943bedf-bundle" (OuterVolumeSpecName: "bundle") pod "171a4baa-eb9a-4048-a9f2-c8561943bedf" (UID: "171a4baa-eb9a-4048-a9f2-c8561943bedf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:23:54.807909 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.807871 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93aa8f77-82c3-4942-b0b7-fb60db1592fb-bundle" (OuterVolumeSpecName: "bundle") pod "93aa8f77-82c3-4942-b0b7-fb60db1592fb" (UID: "93aa8f77-82c3-4942-b0b7-fb60db1592fb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:23:54.808096 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.808071 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2453a3-e95b-4d23-88df-22fc028397c4-bundle" (OuterVolumeSpecName: "bundle") pod "ca2453a3-e95b-4d23-88df-22fc028397c4" (UID: "ca2453a3-e95b-4d23-88df-22fc028397c4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:23:54.809218 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.809188 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2453a3-e95b-4d23-88df-22fc028397c4-kube-api-access-dtr76" (OuterVolumeSpecName: "kube-api-access-dtr76") pod "ca2453a3-e95b-4d23-88df-22fc028397c4" (UID: "ca2453a3-e95b-4d23-88df-22fc028397c4"). InnerVolumeSpecName "kube-api-access-dtr76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:23:54.809424 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.809406 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93aa8f77-82c3-4942-b0b7-fb60db1592fb-kube-api-access-rf9vz" (OuterVolumeSpecName: "kube-api-access-rf9vz") pod "93aa8f77-82c3-4942-b0b7-fb60db1592fb" (UID: "93aa8f77-82c3-4942-b0b7-fb60db1592fb"). InnerVolumeSpecName "kube-api-access-rf9vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:23:54.809485 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.809433 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a07a1b-8956-4f86-8c82-92a0cbc47420-kube-api-access-p6l88" (OuterVolumeSpecName: "kube-api-access-p6l88") pod "80a07a1b-8956-4f86-8c82-92a0cbc47420" (UID: "80a07a1b-8956-4f86-8c82-92a0cbc47420"). InnerVolumeSpecName "kube-api-access-p6l88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:23:54.809747 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.809704 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/171a4baa-eb9a-4048-a9f2-c8561943bedf-kube-api-access-s5kgc" (OuterVolumeSpecName: "kube-api-access-s5kgc") pod "171a4baa-eb9a-4048-a9f2-c8561943bedf" (UID: "171a4baa-eb9a-4048-a9f2-c8561943bedf"). InnerVolumeSpecName "kube-api-access-s5kgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:23:54.908415 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.908377 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6l88\" (UniqueName: \"kubernetes.io/projected/80a07a1b-8956-4f86-8c82-92a0cbc47420-kube-api-access-p6l88\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:54.908415 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.908411 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5kgc\" (UniqueName: \"kubernetes.io/projected/171a4baa-eb9a-4048-a9f2-c8561943bedf-kube-api-access-s5kgc\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:54.908608 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.908422 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/171a4baa-eb9a-4048-a9f2-c8561943bedf-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:54.908608 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.908439 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/80a07a1b-8956-4f86-8c82-92a0cbc47420-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:54.908608 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.908448 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf9vz\" (UniqueName: \"kubernetes.io/projected/93aa8f77-82c3-4942-b0b7-fb60db1592fb-kube-api-access-rf9vz\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:54.908608 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.908457 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtr76\" (UniqueName: \"kubernetes.io/projected/ca2453a3-e95b-4d23-88df-22fc028397c4-kube-api-access-dtr76\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:54.908608 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.908465 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/93aa8f77-82c3-4942-b0b7-fb60db1592fb-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:54.908608 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:54.908473 2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ca2453a3-e95b-4d23-88df-22fc028397c4-bundle\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:55.403824 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.403779 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca2453a3-e95b-4d23-88df-22fc028397c4-util" (OuterVolumeSpecName: "util") pod "ca2453a3-e95b-4d23-88df-22fc028397c4" (UID: "ca2453a3-e95b-4d23-88df-22fc028397c4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:23:55.413529 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.413478 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ca2453a3-e95b-4d23-88df-22fc028397c4-util\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:55.421809 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.421774 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/171a4baa-eb9a-4048-a9f2-c8561943bedf-util" (OuterVolumeSpecName: "util") pod "171a4baa-eb9a-4048-a9f2-c8561943bedf" (UID: "171a4baa-eb9a-4048-a9f2-c8561943bedf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:23:55.435551 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.435523 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93aa8f77-82c3-4942-b0b7-fb60db1592fb-util" (OuterVolumeSpecName: "util") pod "93aa8f77-82c3-4942-b0b7-fb60db1592fb" (UID: "93aa8f77-82c3-4942-b0b7-fb60db1592fb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:23:55.445925 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.445900 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80a07a1b-8956-4f86-8c82-92a0cbc47420-util" (OuterVolumeSpecName: "util") pod "80a07a1b-8956-4f86-8c82-92a0cbc47420" (UID: "80a07a1b-8956-4f86-8c82-92a0cbc47420"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:23:55.486246 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.486218 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" Sep 08 14:23:55.486416 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.486213 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/eb753013ef17676ca428dee58520cdfd0d566802382f786ec048d87053kj6pd" event={"ID":"171a4baa-eb9a-4048-a9f2-c8561943bedf","Type":"ContainerDied","Data":"97f6a201ddbfdf139b165e7a467df4c43182e3b1490dbfb0a9d649cfb166fbf6"} Sep 08 14:23:55.486416 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.486341 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97f6a201ddbfdf139b165e7a467df4c43182e3b1490dbfb0a9d649cfb166fbf6" Sep 08 14:23:55.488104 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.488087 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" Sep 08 14:23:55.488203 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.488137 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb44z6z" event={"ID":"ca2453a3-e95b-4d23-88df-22fc028397c4","Type":"ContainerDied","Data":"74cd8856f715e580008de2a346d28138fb8458f75a9dfd4fd8cedb6310877ac1"} Sep 08 14:23:55.488203 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.488168 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74cd8856f715e580008de2a346d28138fb8458f75a9dfd4fd8cedb6310877ac1" Sep 08 14:23:55.490295 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.490271 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" Sep 08 14:23:55.490295 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.490279 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/cd2c7243cb31b10fd1249163c10d5ea652384acf3e15d0664e86d7183ccxmrp" event={"ID":"93aa8f77-82c3-4942-b0b7-fb60db1592fb","Type":"ContainerDied","Data":"aa9ed1884ad74025f992485f8c210229a8102d2ca15e7f9b447a4390c55c99c4"} Sep 08 14:23:55.490497 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.490331 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa9ed1884ad74025f992485f8c210229a8102d2ca15e7f9b447a4390c55c99c4" Sep 08 14:23:55.492468 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.492447 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" event={"ID":"80a07a1b-8956-4f86-8c82-92a0cbc47420","Type":"ContainerDied","Data":"7d4bc3b298017d22ebee06d891e14139b68432a2c79bcd5a781a50a26bad6b22"} Sep 08 14:23:55.492555 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.492483 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d4bc3b298017d22ebee06d891e14139b68432a2c79bcd5a781a50a26bad6b22" Sep 08 14:23:55.492555 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.492486 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/71cfa9a46429353ffda10db9087bedfaf65d54e1add7115bd006bed3a0rvqd9" Sep 08 14:23:55.514391 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.514364 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/93aa8f77-82c3-4942-b0b7-fb60db1592fb-util\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:55.514391 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.514385 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/80a07a1b-8956-4f86-8c82-92a0cbc47420-util\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:23:55.514391 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:23:55.514393 2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/171a4baa-eb9a-4048-a9f2-c8561943bedf-util\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:24:01.109004 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.108965 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-wplpz"] Sep 08 14:24:01.109478 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109418 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93aa8f77-82c3-4942-b0b7-fb60db1592fb" containerName="pull" Sep 08 14:24:01.109478 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109436 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93aa8f77-82c3-4942-b0b7-fb60db1592fb" containerName="pull" Sep 08 14:24:01.109478 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109452 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a07a1b-8956-4f86-8c82-92a0cbc47420" containerName="util" Sep 08 14:24:01.109478 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109460 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a07a1b-8956-4f86-8c82-92a0cbc47420" containerName="util" Sep 08 14:24:01.109478 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109470 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="171a4baa-eb9a-4048-a9f2-c8561943bedf" containerName="extract" Sep 08 14:24:01.109478 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109478 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="171a4baa-eb9a-4048-a9f2-c8561943bedf" containerName="extract" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109491 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a07a1b-8956-4f86-8c82-92a0cbc47420" containerName="extract" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109499 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a07a1b-8956-4f86-8c82-92a0cbc47420" containerName="extract" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109513 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2453a3-e95b-4d23-88df-22fc028397c4" containerName="pull" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109521 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2453a3-e95b-4d23-88df-22fc028397c4" containerName="pull" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109535 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="171a4baa-eb9a-4048-a9f2-c8561943bedf" containerName="pull" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109543 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="171a4baa-eb9a-4048-a9f2-c8561943bedf" containerName="pull" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109553 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93aa8f77-82c3-4942-b0b7-fb60db1592fb" containerName="extract" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109562 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93aa8f77-82c3-4942-b0b7-fb60db1592fb" containerName="extract" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109579 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2453a3-e95b-4d23-88df-22fc028397c4" containerName="util" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109588 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2453a3-e95b-4d23-88df-22fc028397c4" containerName="util" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109596 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a07a1b-8956-4f86-8c82-92a0cbc47420" containerName="pull" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109603 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a07a1b-8956-4f86-8c82-92a0cbc47420" containerName="pull" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109614 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="171a4baa-eb9a-4048-a9f2-c8561943bedf" containerName="util" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109622 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="171a4baa-eb9a-4048-a9f2-c8561943bedf" containerName="util" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109630 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93aa8f77-82c3-4942-b0b7-fb60db1592fb" containerName="util" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109638 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93aa8f77-82c3-4942-b0b7-fb60db1592fb" containerName="util" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109648 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca2453a3-e95b-4d23-88df-22fc028397c4" containerName="extract" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109656 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca2453a3-e95b-4d23-88df-22fc028397c4" containerName="extract" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109769 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a07a1b-8956-4f86-8c82-92a0cbc47420" containerName="extract" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109782 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="171a4baa-eb9a-4048-a9f2-c8561943bedf" containerName="extract" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109795 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca2453a3-e95b-4d23-88df-22fc028397c4" containerName="extract" Sep 08 14:24:01.109869 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.109805 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="93aa8f77-82c3-4942-b0b7-fb60db1592fb" containerName="extract" Sep 08 14:24:01.112839 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.112819 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-wplpz" Sep 08 14:24:01.115173 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.115149 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 08 14:24:01.115272 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.115182 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 08 14:24:01.115272 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.115200 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nk2bp\"" Sep 08 14:24:01.122107 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.122086 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-wplpz"] Sep 08 14:24:01.158465 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.158435 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwl4\" (UniqueName: \"kubernetes.io/projected/83d27644-90c8-4fc3-a559-05754424ed05-kube-api-access-lzwl4\") pod \"authorino-operator-68d88948d8-wplpz\" (UID: \"83d27644-90c8-4fc3-a559-05754424ed05\") " pod="kuadrant-system/authorino-operator-68d88948d8-wplpz" Sep 08 14:24:01.259549 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.259520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwl4\" (UniqueName: \"kubernetes.io/projected/83d27644-90c8-4fc3-a559-05754424ed05-kube-api-access-lzwl4\") pod \"authorino-operator-68d88948d8-wplpz\" (UID: \"83d27644-90c8-4fc3-a559-05754424ed05\") " pod="kuadrant-system/authorino-operator-68d88948d8-wplpz" Sep 08 14:24:01.276651 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.276628 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwl4\" (UniqueName: \"kubernetes.io/projected/83d27644-90c8-4fc3-a559-05754424ed05-kube-api-access-lzwl4\") pod \"authorino-operator-68d88948d8-wplpz\" (UID: \"83d27644-90c8-4fc3-a559-05754424ed05\") " pod="kuadrant-system/authorino-operator-68d88948d8-wplpz" Sep 08 14:24:01.423718 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.423631 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d88948d8-wplpz" Sep 08 14:24:01.697063 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:01.697038 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d88948d8-wplpz"] Sep 08 14:24:01.699055 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:24:01.699028 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83d27644_90c8_4fc3_a559_05754424ed05.slice/crio-4a2533c7004cb976f64b0634226dafc5f4cad19247b51b572972f7f3774cd73f WatchSource:0}: Error finding container 4a2533c7004cb976f64b0634226dafc5f4cad19247b51b572972f7f3774cd73f: Status 404 returned error can't find the container with id 4a2533c7004cb976f64b0634226dafc5f4cad19247b51b572972f7f3774cd73f Sep 08 14:24:02.528430 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:02.528400 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-wplpz" event={"ID":"83d27644-90c8-4fc3-a559-05754424ed05","Type":"ContainerStarted","Data":"4a2533c7004cb976f64b0634226dafc5f4cad19247b51b572972f7f3774cd73f"} Sep 08 14:24:04.541050 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:04.540968 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d88948d8-wplpz" event={"ID":"83d27644-90c8-4fc3-a559-05754424ed05","Type":"ContainerStarted","Data":"e21cc1e441ac66bad11cac8bfad322cf18101b2da1de1203d267eaacc834b245"} Sep 08 14:24:04.541383 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:04.541078 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d88948d8-wplpz" Sep 08 14:24:04.570993 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:04.570927 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d88948d8-wplpz" podStartSLOduration=1.063426098 podStartE2EDuration="3.570907816s" totalImagesPullingTime="2.507481718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:24:04.567465265 +0000 UTC m=+456.851782372" watchObservedRunningTime="2026-09-08 14:24:04.570907816 +0000 UTC m=+456.855224932" Sep 08 14:24:05.264386 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:05.264352 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv"] Sep 08 14:24:05.267703 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:05.267687 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:05.270109 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:05.270086 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-stv9z\"" Sep 08 14:24:05.281474 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:05.281453 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv"] Sep 08 14:24:05.293074 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:05.293053 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/303e4399-3a50-4d85-a6ad-11a93c0550ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-d6gtv\" (UID: \"303e4399-3a50-4d85-a6ad-11a93c0550ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:05.293180 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:05.293091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgtfw\" (UniqueName: \"kubernetes.io/projected/303e4399-3a50-4d85-a6ad-11a93c0550ce-kube-api-access-pgtfw\") pod \"kuadrant-operator-controller-manager-58fc56757c-d6gtv\" (UID: \"303e4399-3a50-4d85-a6ad-11a93c0550ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:05.393972 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:05.393937 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/303e4399-3a50-4d85-a6ad-11a93c0550ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-d6gtv\" (UID: \"303e4399-3a50-4d85-a6ad-11a93c0550ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:05.394111 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:05.393983 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgtfw\" (UniqueName: \"kubernetes.io/projected/303e4399-3a50-4d85-a6ad-11a93c0550ce-kube-api-access-pgtfw\") pod \"kuadrant-operator-controller-manager-58fc56757c-d6gtv\" (UID: \"303e4399-3a50-4d85-a6ad-11a93c0550ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:05.394320 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:05.394299 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/303e4399-3a50-4d85-a6ad-11a93c0550ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-d6gtv\" (UID: \"303e4399-3a50-4d85-a6ad-11a93c0550ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:05.411378 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:05.411351 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgtfw\" (UniqueName: \"kubernetes.io/projected/303e4399-3a50-4d85-a6ad-11a93c0550ce-kube-api-access-pgtfw\") pod \"kuadrant-operator-controller-manager-58fc56757c-d6gtv\" (UID: \"303e4399-3a50-4d85-a6ad-11a93c0550ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:05.577518 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:05.577437 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:05.863303 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:05.863271 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv"] Sep 08 14:24:05.866049 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:24:05.866018 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod303e4399_3a50_4d85_a6ad_11a93c0550ce.slice/crio-13e0706885206e3166762e899b0c525110e64e6e7357f64bea57ddfa07179f2c WatchSource:0}: Error finding container 13e0706885206e3166762e899b0c525110e64e6e7357f64bea57ddfa07179f2c: Status 404 returned error can't find the container with id 13e0706885206e3166762e899b0c525110e64e6e7357f64bea57ddfa07179f2c Sep 08 14:24:06.552061 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:06.552022 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" event={"ID":"303e4399-3a50-4d85-a6ad-11a93c0550ce","Type":"ContainerStarted","Data":"13e0706885206e3166762e899b0c525110e64e6e7357f64bea57ddfa07179f2c"} Sep 08 14:24:07.588439 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:07.588403 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-jpgwr"] Sep 08 14:24:07.592261 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:07.592234 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jpgwr" Sep 08 14:24:07.594515 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:07.594494 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Sep 08 14:24:07.594774 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:07.594754 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-rgwpp\"" Sep 08 14:24:07.603329 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:07.603304 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-jpgwr"] Sep 08 14:24:07.612813 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:07.612762 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsc58\" (UniqueName: \"kubernetes.io/projected/eceae5fa-0caf-43ce-a5c0-c9b65c745be3-kube-api-access-hsc58\") pod \"dns-operator-controller-manager-5985564776-jpgwr\" (UID: \"eceae5fa-0caf-43ce-a5c0-c9b65c745be3\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-jpgwr" Sep 08 14:24:07.713911 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:07.713879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsc58\" (UniqueName: \"kubernetes.io/projected/eceae5fa-0caf-43ce-a5c0-c9b65c745be3-kube-api-access-hsc58\") pod \"dns-operator-controller-manager-5985564776-jpgwr\" (UID: \"eceae5fa-0caf-43ce-a5c0-c9b65c745be3\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-jpgwr" Sep 08 14:24:07.722668 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:07.722642 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsc58\" (UniqueName: \"kubernetes.io/projected/eceae5fa-0caf-43ce-a5c0-c9b65c745be3-kube-api-access-hsc58\") pod \"dns-operator-controller-manager-5985564776-jpgwr\" (UID: \"eceae5fa-0caf-43ce-a5c0-c9b65c745be3\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-jpgwr" Sep 08 14:24:07.907741 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:07.907649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jpgwr" Sep 08 14:24:08.210637 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:08.210610 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-jpgwr"] Sep 08 14:24:08.212299 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:24:08.212270 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeceae5fa_0caf_43ce_a5c0_c9b65c745be3.slice/crio-309e14fdf90cc3871f7a099de6eb8f79a6e8b0fc8f426520b4823bf2e8e533f5 WatchSource:0}: Error finding container 309e14fdf90cc3871f7a099de6eb8f79a6e8b0fc8f426520b4823bf2e8e533f5: Status 404 returned error can't find the container with id 309e14fdf90cc3871f7a099de6eb8f79a6e8b0fc8f426520b4823bf2e8e533f5 Sep 08 14:24:08.565668 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:08.565584 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jpgwr" event={"ID":"eceae5fa-0caf-43ce-a5c0-c9b65c745be3","Type":"ContainerStarted","Data":"309e14fdf90cc3871f7a099de6eb8f79a6e8b0fc8f426520b4823bf2e8e533f5"} Sep 08 14:24:10.578650 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:10.578603 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" event={"ID":"303e4399-3a50-4d85-a6ad-11a93c0550ce","Type":"ContainerStarted","Data":"53c4d88d07a9766c85d9fc1b80537dd5e15e1023ac9fd230b71caa3d9d545993"} Sep 08 14:24:10.579183 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:10.578697 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:10.606221 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:10.606174 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" podStartSLOduration=1.537569787 podStartE2EDuration="5.606155024s" totalImagesPullingTime="4.068585237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:24:10.603488558 +0000 UTC m=+462.887805697" watchObservedRunningTime="2026-09-08 14:24:10.606155024 +0000 UTC m=+462.890472143" Sep 08 14:24:11.587325 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:11.587282 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jpgwr" event={"ID":"eceae5fa-0caf-43ce-a5c0-c9b65c745be3","Type":"ContainerStarted","Data":"104e8a8a951b3f08a09d41ca60e4ea85ec1f30989c1d926d1beef5bfddf343b9"} Sep 08 14:24:11.604274 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:11.604217 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jpgwr" podStartSLOduration=2.062632137 podStartE2EDuration="4.604197054s" totalImagesPullingTime="2.541564917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:24:11.603620216 +0000 UTC m=+463.887937332" watchObservedRunningTime="2026-09-08 14:24:11.604197054 +0000 UTC m=+463.888514166" Sep 08 14:24:12.592347 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:12.592316 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jpgwr" Sep 08 14:24:15.549150 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:15.549121 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d88948d8-wplpz" Sep 08 14:24:21.589582 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:21.589553 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:23.196442 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.196400 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54"] Sep 08 14:24:23.199882 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.199859 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:23.216243 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.216214 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54"] Sep 08 14:24:23.251186 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.251155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d7bb936-eb98-4720-bed5-14ba4c4bd071-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-cvj54\" (UID: \"5d7bb936-eb98-4720-bed5-14ba4c4bd071\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:23.251186 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.251186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzdjd\" (UniqueName: \"kubernetes.io/projected/5d7bb936-eb98-4720-bed5-14ba4c4bd071-kube-api-access-qzdjd\") pod \"kuadrant-operator-controller-manager-68cf589589-cvj54\" (UID: \"5d7bb936-eb98-4720-bed5-14ba4c4bd071\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:23.352209 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.352173 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d7bb936-eb98-4720-bed5-14ba4c4bd071-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-cvj54\" (UID: \"5d7bb936-eb98-4720-bed5-14ba4c4bd071\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:23.352209 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.352207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzdjd\" (UniqueName: \"kubernetes.io/projected/5d7bb936-eb98-4720-bed5-14ba4c4bd071-kube-api-access-qzdjd\") pod \"kuadrant-operator-controller-manager-68cf589589-cvj54\" (UID: \"5d7bb936-eb98-4720-bed5-14ba4c4bd071\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:23.352561 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.352542 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d7bb936-eb98-4720-bed5-14ba4c4bd071-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-cvj54\" (UID: \"5d7bb936-eb98-4720-bed5-14ba4c4bd071\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:23.361119 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.361093 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzdjd\" (UniqueName: \"kubernetes.io/projected/5d7bb936-eb98-4720-bed5-14ba4c4bd071-kube-api-access-qzdjd\") pod \"kuadrant-operator-controller-manager-68cf589589-cvj54\" (UID: \"5d7bb936-eb98-4720-bed5-14ba4c4bd071\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:23.509410 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.509383 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:23.598807 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.598778 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jpgwr" Sep 08 14:24:23.669426 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.669399 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv"] Sep 08 14:24:23.669613 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.669590 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" podUID="303e4399-3a50-4d85-a6ad-11a93c0550ce" containerName="manager" containerID="cri-o://53c4d88d07a9766c85d9fc1b80537dd5e15e1023ac9fd230b71caa3d9d545993" gracePeriod=2 Sep 08 14:24:23.671769 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:24:23.671708 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-d6gtv\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="303e4399-3a50-4d85-a6ad-11a93c0550ce" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:23.673317 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.673293 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv"] Sep 08 14:24:23.691792 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.690827 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54"] Sep 08 14:24:23.695849 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.695822 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv"] Sep 08 14:24:23.696225 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.696208 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="303e4399-3a50-4d85-a6ad-11a93c0550ce" containerName="manager" Sep 08 14:24:23.696268 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.696227 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="303e4399-3a50-4d85-a6ad-11a93c0550ce" containerName="manager" Sep 08 14:24:23.696317 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.696306 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="303e4399-3a50-4d85-a6ad-11a93c0550ce" containerName="manager" Sep 08 14:24:23.699625 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.699606 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" Sep 08 14:24:23.699833 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.699809 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54"] Sep 08 14:24:23.713014 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.712991 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg"] Sep 08 14:24:23.716815 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.716794 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" Sep 08 14:24:23.717369 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.717347 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv"] Sep 08 14:24:23.733410 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:24:23.733381 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-d6gtv\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="303e4399-3a50-4d85-a6ad-11a93c0550ce" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:23.744445 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.744422 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg"] Sep 08 14:24:23.756819 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.756786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-664pf\" (UniqueName: \"kubernetes.io/projected/a9282a8e-113f-4d26-92c0-b04ba7c4074e-kube-api-access-664pf\") pod \"kuadrant-operator-controller-manager-58fc56757c-bwnkv\" (UID: \"a9282a8e-113f-4d26-92c0-b04ba7c4074e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" Sep 08 14:24:23.757029 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:24:23.756949 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-d6gtv\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="303e4399-3a50-4d85-a6ad-11a93c0550ce" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:23.757135 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.757034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9282a8e-113f-4d26-92c0-b04ba7c4074e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-bwnkv\" (UID: \"a9282a8e-113f-4d26-92c0-b04ba7c4074e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" Sep 08 14:24:23.858030 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.857999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9282a8e-113f-4d26-92c0-b04ba7c4074e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-bwnkv\" (UID: \"a9282a8e-113f-4d26-92c0-b04ba7c4074e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" Sep 08 14:24:23.858195 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.858036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48de116f-f090-4a6b-bbf6-3bf445fc1f03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-7qzcg\" (UID: \"48de116f-f090-4a6b-bbf6-3bf445fc1f03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" Sep 08 14:24:23.858195 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.858092 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-664pf\" (UniqueName: \"kubernetes.io/projected/a9282a8e-113f-4d26-92c0-b04ba7c4074e-kube-api-access-664pf\") pod \"kuadrant-operator-controller-manager-58fc56757c-bwnkv\" (UID: \"a9282a8e-113f-4d26-92c0-b04ba7c4074e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" Sep 08 14:24:23.858195 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.858127 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thfqc\" (UniqueName: \"kubernetes.io/projected/48de116f-f090-4a6b-bbf6-3bf445fc1f03-kube-api-access-thfqc\") pod \"kuadrant-operator-controller-manager-68cf589589-7qzcg\" (UID: \"48de116f-f090-4a6b-bbf6-3bf445fc1f03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" Sep 08 14:24:23.858387 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.858364 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9282a8e-113f-4d26-92c0-b04ba7c4074e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-bwnkv\" (UID: \"a9282a8e-113f-4d26-92c0-b04ba7c4074e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" Sep 08 14:24:23.868131 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.868103 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-664pf\" (UniqueName: \"kubernetes.io/projected/a9282a8e-113f-4d26-92c0-b04ba7c4074e-kube-api-access-664pf\") pod \"kuadrant-operator-controller-manager-58fc56757c-bwnkv\" (UID: \"a9282a8e-113f-4d26-92c0-b04ba7c4074e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" Sep 08 14:24:23.893439 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:24:23.893410 2578 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 08 14:24:23.893439 ip-10-0-141-155 kubenswrapper[2578]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-68cf589589-cvj54_kuadrant-system_5d7bb936-eb98-4720-bed5-14ba4c4bd071_0(98bd22828aab03eaa0ae33b88d6f6352899482e48ab593cdabb486dbf893a043): error adding pod kuadrant-system_kuadrant-operator-controller-manager-68cf589589-cvj54 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"98bd22828aab03eaa0ae33b88d6f6352899482e48ab593cdabb486dbf893a043" Netns:"/var/run/netns/ebe6f035-5861-475e-bc53-727d57410304" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-68cf589589-cvj54;K8S_POD_INFRA_CONTAINER_ID=98bd22828aab03eaa0ae33b88d6f6352899482e48ab593cdabb486dbf893a043;K8S_POD_UID=5d7bb936-eb98-4720-bed5-14ba4c4bd071" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54] networking: [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54/5d7bb936-eb98-4720-bed5-14ba4c4bd071:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54 98bd22828aab03eaa0ae33b88d6f6352899482e48ab593cdabb486dbf893a043 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:85:00:2c [10.133.0.44/23] Sep 08 14:24:23.893439 ip-10-0-141-155 kubenswrapper[2578]: ' Sep 08 14:24:23.893439 ip-10-0-141-155 kubenswrapper[2578]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 08 14:24:23.893439 ip-10-0-141-155 kubenswrapper[2578]: > Sep 08 14:24:23.893869 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:24:23.893486 2578 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 08 14:24:23.893869 ip-10-0-141-155 kubenswrapper[2578]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-68cf589589-cvj54_kuadrant-system_5d7bb936-eb98-4720-bed5-14ba4c4bd071_0(98bd22828aab03eaa0ae33b88d6f6352899482e48ab593cdabb486dbf893a043): error adding pod kuadrant-system_kuadrant-operator-controller-manager-68cf589589-cvj54 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"98bd22828aab03eaa0ae33b88d6f6352899482e48ab593cdabb486dbf893a043" Netns:"/var/run/netns/ebe6f035-5861-475e-bc53-727d57410304" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-68cf589589-cvj54;K8S_POD_INFRA_CONTAINER_ID=98bd22828aab03eaa0ae33b88d6f6352899482e48ab593cdabb486dbf893a043;K8S_POD_UID=5d7bb936-eb98-4720-bed5-14ba4c4bd071" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54] networking: [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54/5d7bb936-eb98-4720-bed5-14ba4c4bd071:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54 98bd22828aab03eaa0ae33b88d6f6352899482e48ab593cdabb486dbf893a043 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:85:00:2c [10.133.0.44/23] Sep 08 14:24:23.893869 ip-10-0-141-155 kubenswrapper[2578]: ' Sep 08 14:24:23.893869 ip-10-0-141-155 kubenswrapper[2578]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 08 14:24:23.893869 ip-10-0-141-155 kubenswrapper[2578]: > pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:23.898062 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.898042 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:23.900003 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:24:23.899982 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-58fc56757c-d6gtv\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="303e4399-3a50-4d85-a6ad-11a93c0550ce" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:23.959020 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.958994 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/303e4399-3a50-4d85-a6ad-11a93c0550ce-extensions-socket-volume\") pod \"303e4399-3a50-4d85-a6ad-11a93c0550ce\" (UID: \"303e4399-3a50-4d85-a6ad-11a93c0550ce\") " Sep 08 14:24:23.959122 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.959031 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgtfw\" (UniqueName: \"kubernetes.io/projected/303e4399-3a50-4d85-a6ad-11a93c0550ce-kube-api-access-pgtfw\") pod \"303e4399-3a50-4d85-a6ad-11a93c0550ce\" (UID: \"303e4399-3a50-4d85-a6ad-11a93c0550ce\") " Sep 08 14:24:23.959299 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.959279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thfqc\" (UniqueName: \"kubernetes.io/projected/48de116f-f090-4a6b-bbf6-3bf445fc1f03-kube-api-access-thfqc\") pod \"kuadrant-operator-controller-manager-68cf589589-7qzcg\" (UID: \"48de116f-f090-4a6b-bbf6-3bf445fc1f03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" Sep 08 14:24:23.959371 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.959344 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48de116f-f090-4a6b-bbf6-3bf445fc1f03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-7qzcg\" (UID: \"48de116f-f090-4a6b-bbf6-3bf445fc1f03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" Sep 08 14:24:23.959524 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.959501 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/303e4399-3a50-4d85-a6ad-11a93c0550ce-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "303e4399-3a50-4d85-a6ad-11a93c0550ce" (UID: "303e4399-3a50-4d85-a6ad-11a93c0550ce"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:24:23.959717 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.959699 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48de116f-f090-4a6b-bbf6-3bf445fc1f03-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-7qzcg\" (UID: \"48de116f-f090-4a6b-bbf6-3bf445fc1f03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" Sep 08 14:24:23.961376 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.961356 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/303e4399-3a50-4d85-a6ad-11a93c0550ce-kube-api-access-pgtfw" (OuterVolumeSpecName: "kube-api-access-pgtfw") pod "303e4399-3a50-4d85-a6ad-11a93c0550ce" (UID: "303e4399-3a50-4d85-a6ad-11a93c0550ce"). InnerVolumeSpecName "kube-api-access-pgtfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:24:23.971509 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:23.971490 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thfqc\" (UniqueName: \"kubernetes.io/projected/48de116f-f090-4a6b-bbf6-3bf445fc1f03-kube-api-access-thfqc\") pod \"kuadrant-operator-controller-manager-68cf589589-7qzcg\" (UID: \"48de116f-f090-4a6b-bbf6-3bf445fc1f03\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" Sep 08 14:24:24.054218 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.054149 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" Sep 08 14:24:24.060357 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.060342 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/303e4399-3a50-4d85-a6ad-11a93c0550ce-extensions-socket-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:24:24.060416 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.060360 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgtfw\" (UniqueName: \"kubernetes.io/projected/303e4399-3a50-4d85-a6ad-11a93c0550ce-kube-api-access-pgtfw\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:24:24.061228 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.061213 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" Sep 08 14:24:24.241426 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.241396 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="303e4399-3a50-4d85-a6ad-11a93c0550ce" path="/var/lib/kubelet/pods/303e4399-3a50-4d85-a6ad-11a93c0550ce/volumes" Sep 08 14:24:24.334011 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.333981 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv"] Sep 08 14:24:24.334908 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:24:24.334884 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9282a8e_113f_4d26_92c0_b04ba7c4074e.slice/crio-8be1897331c59f7fd1f5a88fe6fa02e68e5a4d9b71811d7bc226125cf0c6c0ae WatchSource:0}: Error finding container 8be1897331c59f7fd1f5a88fe6fa02e68e5a4d9b71811d7bc226125cf0c6c0ae: Status 404 returned error can't find the container with id 8be1897331c59f7fd1f5a88fe6fa02e68e5a4d9b71811d7bc226125cf0c6c0ae Sep 08 14:24:24.336605 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.336583 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg"] Sep 08 14:24:24.337959 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:24:24.337935 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48de116f_f090_4a6b_bbf6_3bf445fc1f03.slice/crio-f6d1ddeacd9d64d34f93365b2e8dc75dfadb87b68a1bfad16c102b7eeb70552b WatchSource:0}: Error finding container f6d1ddeacd9d64d34f93365b2e8dc75dfadb87b68a1bfad16c102b7eeb70552b: Status 404 returned error can't find the container with id f6d1ddeacd9d64d34f93365b2e8dc75dfadb87b68a1bfad16c102b7eeb70552b Sep 08 14:24:24.666196 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.666106 2578 generic.go:356] "Generic (PLEG): container finished" podID="303e4399-3a50-4d85-a6ad-11a93c0550ce" containerID="53c4d88d07a9766c85d9fc1b80537dd5e15e1023ac9fd230b71caa3d9d545993" exitCode=0 Sep 08 14:24:24.666196 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.666158 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-d6gtv" Sep 08 14:24:24.666395 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.666211 2578 scope.go:117] "RemoveContainer" containerID="53c4d88d07a9766c85d9fc1b80537dd5e15e1023ac9fd230b71caa3d9d545993" Sep 08 14:24:24.668256 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.668226 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" event={"ID":"a9282a8e-113f-4d26-92c0-b04ba7c4074e","Type":"ContainerStarted","Data":"e5b41c84614604b441e60f67ebc794ab20998fda4d1e61d0de67705bf7a26618"} Sep 08 14:24:24.668420 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.668317 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" Sep 08 14:24:24.668420 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.668338 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" event={"ID":"a9282a8e-113f-4d26-92c0-b04ba7c4074e","Type":"ContainerStarted","Data":"8be1897331c59f7fd1f5a88fe6fa02e68e5a4d9b71811d7bc226125cf0c6c0ae"} Sep 08 14:24:24.670243 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.670218 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" event={"ID":"48de116f-f090-4a6b-bbf6-3bf445fc1f03","Type":"ContainerStarted","Data":"83d01a9ef5ae932d1f90a72527638fd84952c62a42be4b88dbf49e66ced701fb"} Sep 08 14:24:24.670243 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.670237 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:24.670386 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.670259 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" event={"ID":"48de116f-f090-4a6b-bbf6-3bf445fc1f03","Type":"ContainerStarted","Data":"f6d1ddeacd9d64d34f93365b2e8dc75dfadb87b68a1bfad16c102b7eeb70552b"} Sep 08 14:24:24.677585 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.677564 2578 scope.go:117] "RemoveContainer" containerID="53c4d88d07a9766c85d9fc1b80537dd5e15e1023ac9fd230b71caa3d9d545993" Sep 08 14:24:24.677924 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:24:24.677903 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c4d88d07a9766c85d9fc1b80537dd5e15e1023ac9fd230b71caa3d9d545993\": container with ID starting with 53c4d88d07a9766c85d9fc1b80537dd5e15e1023ac9fd230b71caa3d9d545993 not found: ID does not exist" containerID="53c4d88d07a9766c85d9fc1b80537dd5e15e1023ac9fd230b71caa3d9d545993" Sep 08 14:24:24.678005 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.677934 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c4d88d07a9766c85d9fc1b80537dd5e15e1023ac9fd230b71caa3d9d545993"} err="failed to get container status \"53c4d88d07a9766c85d9fc1b80537dd5e15e1023ac9fd230b71caa3d9d545993\": rpc error: code = NotFound desc = could not find container \"53c4d88d07a9766c85d9fc1b80537dd5e15e1023ac9fd230b71caa3d9d545993\": container with ID starting with 53c4d88d07a9766c85d9fc1b80537dd5e15e1023ac9fd230b71caa3d9d545993 not found: ID does not exist" Sep 08 14:24:24.679217 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.679198 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:24.689384 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.689340 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" podStartSLOduration=1.689326598 podStartE2EDuration="1.689326598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:24:24.686097267 +0000 UTC m=+476.970414422" watchObservedRunningTime="2026-09-08 14:24:24.689326598 +0000 UTC m=+476.973643713" Sep 08 14:24:24.711320 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:24:24.711280 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-cvj54\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="5d7bb936-eb98-4720-bed5-14ba4c4bd071" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:24.712183 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.712144 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" podStartSLOduration=1.712130937 podStartE2EDuration="1.712130937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:24:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:24:24.709396557 +0000 UTC m=+476.993713672" watchObservedRunningTime="2026-09-08 14:24:24.712130937 +0000 UTC m=+476.996448052" Sep 08 14:24:24.765611 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.765587 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d7bb936-eb98-4720-bed5-14ba4c4bd071-extensions-socket-volume\") pod \"5d7bb936-eb98-4720-bed5-14ba4c4bd071\" (UID: \"5d7bb936-eb98-4720-bed5-14ba4c4bd071\") " Sep 08 14:24:24.765776 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.765693 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzdjd\" (UniqueName: \"kubernetes.io/projected/5d7bb936-eb98-4720-bed5-14ba4c4bd071-kube-api-access-qzdjd\") pod \"5d7bb936-eb98-4720-bed5-14ba4c4bd071\" (UID: \"5d7bb936-eb98-4720-bed5-14ba4c4bd071\") " Sep 08 14:24:24.765855 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.765833 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7bb936-eb98-4720-bed5-14ba4c4bd071-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5d7bb936-eb98-4720-bed5-14ba4c4bd071" (UID: "5d7bb936-eb98-4720-bed5-14ba4c4bd071"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:24:24.766003 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.765987 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d7bb936-eb98-4720-bed5-14ba4c4bd071-extensions-socket-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:24:24.767805 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.767784 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7bb936-eb98-4720-bed5-14ba4c4bd071-kube-api-access-qzdjd" (OuterVolumeSpecName: "kube-api-access-qzdjd") pod "5d7bb936-eb98-4720-bed5-14ba4c4bd071" (UID: "5d7bb936-eb98-4720-bed5-14ba4c4bd071"). InnerVolumeSpecName "kube-api-access-qzdjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:24:24.866629 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:24.866588 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzdjd\" (UniqueName: \"kubernetes.io/projected/5d7bb936-eb98-4720-bed5-14ba4c4bd071-kube-api-access-qzdjd\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:24:25.677076 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:25.677048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:25.677447 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:25.677178 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" Sep 08 14:24:25.679113 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:24:25.679090 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-cvj54\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="5d7bb936-eb98-4720-bed5-14ba4c4bd071" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:25.685167 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:24:25.685136 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-cvj54\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="5d7bb936-eb98-4720-bed5-14ba4c4bd071" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-cvj54" Sep 08 14:24:26.241524 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:26.241481 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d7bb936-eb98-4720-bed5-14ba4c4bd071" path="/var/lib/kubelet/pods/5d7bb936-eb98-4720-bed5-14ba4c4bd071/volumes" Sep 08 14:24:35.679410 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:35.679380 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" Sep 08 14:24:36.684883 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:36.684847 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" Sep 08 14:24:36.742346 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:36.742316 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv"] Sep 08 14:24:36.742554 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:36.742532 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" podUID="a9282a8e-113f-4d26-92c0-b04ba7c4074e" containerName="manager" containerID="cri-o://e5b41c84614604b441e60f67ebc794ab20998fda4d1e61d0de67705bf7a26618" gracePeriod=10 Sep 08 14:24:36.992283 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:36.992258 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" Sep 08 14:24:37.071127 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.071099 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9282a8e-113f-4d26-92c0-b04ba7c4074e-extensions-socket-volume\") pod \"a9282a8e-113f-4d26-92c0-b04ba7c4074e\" (UID: \"a9282a8e-113f-4d26-92c0-b04ba7c4074e\") " Sep 08 14:24:37.071314 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.071178 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-664pf\" (UniqueName: \"kubernetes.io/projected/a9282a8e-113f-4d26-92c0-b04ba7c4074e-kube-api-access-664pf\") pod \"a9282a8e-113f-4d26-92c0-b04ba7c4074e\" (UID: \"a9282a8e-113f-4d26-92c0-b04ba7c4074e\") " Sep 08 14:24:37.071494 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.071469 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9282a8e-113f-4d26-92c0-b04ba7c4074e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a9282a8e-113f-4d26-92c0-b04ba7c4074e" (UID: "a9282a8e-113f-4d26-92c0-b04ba7c4074e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:24:37.073217 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.073191 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9282a8e-113f-4d26-92c0-b04ba7c4074e-kube-api-access-664pf" (OuterVolumeSpecName: "kube-api-access-664pf") pod "a9282a8e-113f-4d26-92c0-b04ba7c4074e" (UID: "a9282a8e-113f-4d26-92c0-b04ba7c4074e"). InnerVolumeSpecName "kube-api-access-664pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:24:37.172116 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.172071 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a9282a8e-113f-4d26-92c0-b04ba7c4074e-extensions-socket-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:24:37.172116 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.172109 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-664pf\" (UniqueName: \"kubernetes.io/projected/a9282a8e-113f-4d26-92c0-b04ba7c4074e-kube-api-access-664pf\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:24:37.740941 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.740902 2578 generic.go:356] "Generic (PLEG): container finished" podID="a9282a8e-113f-4d26-92c0-b04ba7c4074e" containerID="e5b41c84614604b441e60f67ebc794ab20998fda4d1e61d0de67705bf7a26618" exitCode=0 Sep 08 14:24:37.741326 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.740949 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" Sep 08 14:24:37.741326 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.740997 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" event={"ID":"a9282a8e-113f-4d26-92c0-b04ba7c4074e","Type":"ContainerDied","Data":"e5b41c84614604b441e60f67ebc794ab20998fda4d1e61d0de67705bf7a26618"} Sep 08 14:24:37.741326 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.741036 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv" event={"ID":"a9282a8e-113f-4d26-92c0-b04ba7c4074e","Type":"ContainerDied","Data":"8be1897331c59f7fd1f5a88fe6fa02e68e5a4d9b71811d7bc226125cf0c6c0ae"} Sep 08 14:24:37.741326 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.741057 2578 scope.go:117] "RemoveContainer" containerID="e5b41c84614604b441e60f67ebc794ab20998fda4d1e61d0de67705bf7a26618" Sep 08 14:24:37.750443 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.750222 2578 scope.go:117] "RemoveContainer" containerID="e5b41c84614604b441e60f67ebc794ab20998fda4d1e61d0de67705bf7a26618" Sep 08 14:24:37.750511 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:24:37.750493 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b41c84614604b441e60f67ebc794ab20998fda4d1e61d0de67705bf7a26618\": container with ID starting with e5b41c84614604b441e60f67ebc794ab20998fda4d1e61d0de67705bf7a26618 not found: ID does not exist" containerID="e5b41c84614604b441e60f67ebc794ab20998fda4d1e61d0de67705bf7a26618" Sep 08 14:24:37.750556 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.750520 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b41c84614604b441e60f67ebc794ab20998fda4d1e61d0de67705bf7a26618"} err="failed to get container status \"e5b41c84614604b441e60f67ebc794ab20998fda4d1e61d0de67705bf7a26618\": rpc error: code = NotFound desc = could not find container \"e5b41c84614604b441e60f67ebc794ab20998fda4d1e61d0de67705bf7a26618\": container with ID starting with e5b41c84614604b441e60f67ebc794ab20998fda4d1e61d0de67705bf7a26618 not found: ID does not exist" Sep 08 14:24:37.763477 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.763452 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv"] Sep 08 14:24:37.771528 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:37.771498 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-bwnkv"] Sep 08 14:24:38.241092 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:38.241065 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9282a8e-113f-4d26-92c0-b04ba7c4074e" path="/var/lib/kubelet/pods/a9282a8e-113f-4d26-92c0-b04ba7c4074e/volumes" Sep 08 14:24:53.130062 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.130032 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz"] Sep 08 14:24:53.130493 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.130362 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9282a8e-113f-4d26-92c0-b04ba7c4074e" containerName="manager" Sep 08 14:24:53.130493 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.130373 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9282a8e-113f-4d26-92c0-b04ba7c4074e" containerName="manager" Sep 08 14:24:53.130493 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.130444 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9282a8e-113f-4d26-92c0-b04ba7c4074e" containerName="manager" Sep 08 14:24:53.132638 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.132609 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.134934 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.134912 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 08 14:24:53.135227 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.135207 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 08 14:24:53.135335 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.135212 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 08 14:24:53.135335 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.135253 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-65jgf\"" Sep 08 14:24:53.145410 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.145384 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz"] Sep 08 14:24:53.150835 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.150813 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg"] Sep 08 14:24:53.151039 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.151001 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" podUID="48de116f-f090-4a6b-bbf6-3bf445fc1f03" containerName="manager" containerID="cri-o://83d01a9ef5ae932d1f90a72527638fd84952c62a42be4b88dbf49e66ced701fb" gracePeriod=10 Sep 08 14:24:53.207903 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.207865 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ac3b36-1d40-4ebe-a52f-31237baa182e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.207903 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.207898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.208127 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.207916 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ac3b36-1d40-4ebe-a52f-31237baa182e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.208127 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.207993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.208127 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.208042 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.208127 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.208060 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ac3b36-1d40-4ebe-a52f-31237baa182e-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.208127 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.208078 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.208368 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.208181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.208368 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.208219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d9z7\" (UniqueName: \"kubernetes.io/projected/07ac3b36-1d40-4ebe-a52f-31237baa182e-kube-api-access-4d9z7\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.309839 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.309803 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.310047 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.310028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d9z7\" (UniqueName: \"kubernetes.io/projected/07ac3b36-1d40-4ebe-a52f-31237baa182e-kube-api-access-4d9z7\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.310186 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.310172 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ac3b36-1d40-4ebe-a52f-31237baa182e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.310290 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.310278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.310386 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.310368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ac3b36-1d40-4ebe-a52f-31237baa182e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.310490 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.310479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.310625 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.310611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.310718 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.310707 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ac3b36-1d40-4ebe-a52f-31237baa182e-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.310857 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.310842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.311352 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.311331 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.311695 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.311677 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.312764 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.312743 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07ac3b36-1d40-4ebe-a52f-31237baa182e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.313176 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.313157 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.316397 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.316372 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.316536 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.316520 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07ac3b36-1d40-4ebe-a52f-31237baa182e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.318278 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.318258 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07ac3b36-1d40-4ebe-a52f-31237baa182e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.325303 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.325280 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d9z7\" (UniqueName: \"kubernetes.io/projected/07ac3b36-1d40-4ebe-a52f-31237baa182e-kube-api-access-4d9z7\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.325515 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.325499 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07ac3b36-1d40-4ebe-a52f-31237baa182e-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-jwqsz\" (UID: \"07ac3b36-1d40-4ebe-a52f-31237baa182e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.446055 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.446026 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:53.480583 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.480561 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" Sep 08 14:24:53.512814 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.512778 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thfqc\" (UniqueName: \"kubernetes.io/projected/48de116f-f090-4a6b-bbf6-3bf445fc1f03-kube-api-access-thfqc\") pod \"48de116f-f090-4a6b-bbf6-3bf445fc1f03\" (UID: \"48de116f-f090-4a6b-bbf6-3bf445fc1f03\") " Sep 08 14:24:53.513543 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.513525 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48de116f-f090-4a6b-bbf6-3bf445fc1f03-extensions-socket-volume\") pod \"48de116f-f090-4a6b-bbf6-3bf445fc1f03\" (UID: \"48de116f-f090-4a6b-bbf6-3bf445fc1f03\") " Sep 08 14:24:53.514182 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.514142 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48de116f-f090-4a6b-bbf6-3bf445fc1f03-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "48de116f-f090-4a6b-bbf6-3bf445fc1f03" (UID: "48de116f-f090-4a6b-bbf6-3bf445fc1f03"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 08 14:24:53.516433 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.516414 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48de116f-f090-4a6b-bbf6-3bf445fc1f03-kube-api-access-thfqc" (OuterVolumeSpecName: "kube-api-access-thfqc") pod "48de116f-f090-4a6b-bbf6-3bf445fc1f03" (UID: "48de116f-f090-4a6b-bbf6-3bf445fc1f03"). InnerVolumeSpecName "kube-api-access-thfqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:24:53.614977 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.614944 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thfqc\" (UniqueName: \"kubernetes.io/projected/48de116f-f090-4a6b-bbf6-3bf445fc1f03-kube-api-access-thfqc\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:24:53.614977 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.614970 2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/48de116f-f090-4a6b-bbf6-3bf445fc1f03-extensions-socket-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:24:53.723637 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.723612 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz"] Sep 08 14:24:53.725173 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:24:53.725147 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ac3b36_1d40_4ebe_a52f_31237baa182e.slice/crio-60e4a52d8782dd11ebff1ae6a1315959f377354e1feb69a5bbd0e028b1b0f79f WatchSource:0}: Error finding container 60e4a52d8782dd11ebff1ae6a1315959f377354e1feb69a5bbd0e028b1b0f79f: Status 404 returned error can't find the container with id 60e4a52d8782dd11ebff1ae6a1315959f377354e1feb69a5bbd0e028b1b0f79f Sep 08 14:24:53.826973 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.826936 2578 generic.go:356] "Generic (PLEG): container finished" podID="48de116f-f090-4a6b-bbf6-3bf445fc1f03" containerID="83d01a9ef5ae932d1f90a72527638fd84952c62a42be4b88dbf49e66ced701fb" exitCode=0 Sep 08 14:24:53.827142 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.826981 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" Sep 08 14:24:53.827142 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.827027 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" event={"ID":"48de116f-f090-4a6b-bbf6-3bf445fc1f03","Type":"ContainerDied","Data":"83d01a9ef5ae932d1f90a72527638fd84952c62a42be4b88dbf49e66ced701fb"} Sep 08 14:24:53.827142 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.827086 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg" event={"ID":"48de116f-f090-4a6b-bbf6-3bf445fc1f03","Type":"ContainerDied","Data":"f6d1ddeacd9d64d34f93365b2e8dc75dfadb87b68a1bfad16c102b7eeb70552b"} Sep 08 14:24:53.827142 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.827119 2578 scope.go:117] "RemoveContainer" containerID="83d01a9ef5ae932d1f90a72527638fd84952c62a42be4b88dbf49e66ced701fb" Sep 08 14:24:53.828524 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.828500 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" event={"ID":"07ac3b36-1d40-4ebe-a52f-31237baa182e","Type":"ContainerStarted","Data":"60e4a52d8782dd11ebff1ae6a1315959f377354e1feb69a5bbd0e028b1b0f79f"} Sep 08 14:24:53.836585 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.836571 2578 scope.go:117] "RemoveContainer" containerID="83d01a9ef5ae932d1f90a72527638fd84952c62a42be4b88dbf49e66ced701fb" Sep 08 14:24:53.836907 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:24:53.836890 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d01a9ef5ae932d1f90a72527638fd84952c62a42be4b88dbf49e66ced701fb\": container with ID starting with 83d01a9ef5ae932d1f90a72527638fd84952c62a42be4b88dbf49e66ced701fb not found: ID does not exist" containerID="83d01a9ef5ae932d1f90a72527638fd84952c62a42be4b88dbf49e66ced701fb" Sep 08 14:24:53.836962 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.836915 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d01a9ef5ae932d1f90a72527638fd84952c62a42be4b88dbf49e66ced701fb"} err="failed to get container status \"83d01a9ef5ae932d1f90a72527638fd84952c62a42be4b88dbf49e66ced701fb\": rpc error: code = NotFound desc = could not find container \"83d01a9ef5ae932d1f90a72527638fd84952c62a42be4b88dbf49e66ced701fb\": container with ID starting with 83d01a9ef5ae932d1f90a72527638fd84952c62a42be4b88dbf49e66ced701fb not found: ID does not exist" Sep 08 14:24:53.856398 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.856363 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg"] Sep 08 14:24:53.858520 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:53.858502 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-7qzcg"] Sep 08 14:24:54.242131 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:54.242102 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48de116f-f090-4a6b-bbf6-3bf445fc1f03" path="/var/lib/kubelet/pods/48de116f-f090-4a6b-bbf6-3bf445fc1f03/volumes" Sep 08 14:24:58.594239 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:58.594204 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 08 14:24:58.594602 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:58.594292 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 08 14:24:58.594602 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:58.594338 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 08 14:24:58.863329 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:58.863244 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" event={"ID":"07ac3b36-1d40-4ebe-a52f-31237baa182e","Type":"ContainerStarted","Data":"39edd21109637bdff419f5e5dc15728612ef7a18d20788998d5b08c41adeda16"} Sep 08 14:24:58.884086 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:58.884039 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" podStartSLOduration=1.017182856 podStartE2EDuration="5.884025232s" totalImagesPullingTime="4.866842376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:24:58.881813984 +0000 UTC m=+511.166131100" watchObservedRunningTime="2026-09-08 14:24:58.884025232 +0000 UTC m=+511.168342346" Sep 08 14:24:59.447107 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:59.447075 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:59.451815 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:59.451784 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:59.868152 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:59.868119 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:24:59.868967 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:24:59.868950 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:25:11.953792 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:11.953761 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pr89q"] Sep 08 14:25:11.954149 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:11.954097 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48de116f-f090-4a6b-bbf6-3bf445fc1f03" containerName="manager" Sep 08 14:25:11.954149 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:11.954108 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="48de116f-f090-4a6b-bbf6-3bf445fc1f03" containerName="manager" Sep 08 14:25:11.954224 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:11.954165 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="48de116f-f090-4a6b-bbf6-3bf445fc1f03" containerName="manager" Sep 08 14:25:11.956291 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:11.956275 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" Sep 08 14:25:11.958466 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:11.958449 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 08 14:25:11.958533 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:11.958507 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-74m58\"" Sep 08 14:25:11.965829 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:11.965808 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pr89q"] Sep 08 14:25:12.058202 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.058164 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pr89q"] Sep 08 14:25:12.069890 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.069861 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzh7r\" (UniqueName: \"kubernetes.io/projected/79c20f6b-8b14-4b45-baae-d25702439256-kube-api-access-gzh7r\") pod \"limitador-limitador-859c4d96d6-pr89q\" (UID: \"79c20f6b-8b14-4b45-baae-d25702439256\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" Sep 08 14:25:12.070016 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.069896 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79c20f6b-8b14-4b45-baae-d25702439256-config-file\") pod \"limitador-limitador-859c4d96d6-pr89q\" (UID: \"79c20f6b-8b14-4b45-baae-d25702439256\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" Sep 08 14:25:12.171214 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.171183 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzh7r\" (UniqueName: \"kubernetes.io/projected/79c20f6b-8b14-4b45-baae-d25702439256-kube-api-access-gzh7r\") pod \"limitador-limitador-859c4d96d6-pr89q\" (UID: \"79c20f6b-8b14-4b45-baae-d25702439256\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" Sep 08 14:25:12.171214 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.171218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79c20f6b-8b14-4b45-baae-d25702439256-config-file\") pod \"limitador-limitador-859c4d96d6-pr89q\" (UID: \"79c20f6b-8b14-4b45-baae-d25702439256\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" Sep 08 14:25:12.171843 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.171825 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79c20f6b-8b14-4b45-baae-d25702439256-config-file\") pod \"limitador-limitador-859c4d96d6-pr89q\" (UID: \"79c20f6b-8b14-4b45-baae-d25702439256\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" Sep 08 14:25:12.179503 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.179477 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzh7r\" (UniqueName: \"kubernetes.io/projected/79c20f6b-8b14-4b45-baae-d25702439256-kube-api-access-gzh7r\") pod \"limitador-limitador-859c4d96d6-pr89q\" (UID: \"79c20f6b-8b14-4b45-baae-d25702439256\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" Sep 08 14:25:12.267594 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.267567 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" Sep 08 14:25:12.537599 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.537577 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pr89q"] Sep 08 14:25:12.539625 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:25:12.539589 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c20f6b_8b14_4b45_baae_d25702439256.slice/crio-998dae79b4a4077dc85cd1bf8bf99f8ebc73cd04b6bd9d3b6e2a1ae22cad209e WatchSource:0}: Error finding container 998dae79b4a4077dc85cd1bf8bf99f8ebc73cd04b6bd9d3b6e2a1ae22cad209e: Status 404 returned error can't find the container with id 998dae79b4a4077dc85cd1bf8bf99f8ebc73cd04b6bd9d3b6e2a1ae22cad209e Sep 08 14:25:12.783935 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.783904 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-5fcsb"] Sep 08 14:25:12.787230 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.787169 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-5fcsb" Sep 08 14:25:12.789214 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.789189 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-d88kc\"" Sep 08 14:25:12.795659 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.795639 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-5fcsb"] Sep 08 14:25:12.878467 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.878440 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-547hz\" (UniqueName: \"kubernetes.io/projected/2218070e-23a8-49f1-bfe7-d177675b7e7c-kube-api-access-547hz\") pod \"authorino-b54df94b4-5fcsb\" (UID: \"2218070e-23a8-49f1-bfe7-d177675b7e7c\") " pod="kuadrant-system/authorino-b54df94b4-5fcsb" Sep 08 14:25:12.917765 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.917717 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-789f7"] Sep 08 14:25:12.920188 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.920171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-789f7" Sep 08 14:25:12.928634 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.928610 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-789f7"] Sep 08 14:25:12.948078 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.948047 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" event={"ID":"79c20f6b-8b14-4b45-baae-d25702439256","Type":"ContainerStarted","Data":"998dae79b4a4077dc85cd1bf8bf99f8ebc73cd04b6bd9d3b6e2a1ae22cad209e"} Sep 08 14:25:12.979505 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.979468 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96x6j\" (UniqueName: \"kubernetes.io/projected/3675fec3-3794-4fbf-93d2-07fd3f882876-kube-api-access-96x6j\") pod \"authorino-c548b5d7d-789f7\" (UID: \"3675fec3-3794-4fbf-93d2-07fd3f882876\") " pod="kuadrant-system/authorino-c548b5d7d-789f7" Sep 08 14:25:12.979873 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.979562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-547hz\" (UniqueName: \"kubernetes.io/projected/2218070e-23a8-49f1-bfe7-d177675b7e7c-kube-api-access-547hz\") pod \"authorino-b54df94b4-5fcsb\" (UID: \"2218070e-23a8-49f1-bfe7-d177675b7e7c\") " pod="kuadrant-system/authorino-b54df94b4-5fcsb" Sep 08 14:25:12.987463 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:12.987442 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-547hz\" (UniqueName: \"kubernetes.io/projected/2218070e-23a8-49f1-bfe7-d177675b7e7c-kube-api-access-547hz\") pod \"authorino-b54df94b4-5fcsb\" (UID: \"2218070e-23a8-49f1-bfe7-d177675b7e7c\") " pod="kuadrant-system/authorino-b54df94b4-5fcsb" Sep 08 14:25:13.080379 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:13.080304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96x6j\" (UniqueName: \"kubernetes.io/projected/3675fec3-3794-4fbf-93d2-07fd3f882876-kube-api-access-96x6j\") pod \"authorino-c548b5d7d-789f7\" (UID: \"3675fec3-3794-4fbf-93d2-07fd3f882876\") " pod="kuadrant-system/authorino-c548b5d7d-789f7" Sep 08 14:25:13.088885 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:13.088867 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96x6j\" (UniqueName: \"kubernetes.io/projected/3675fec3-3794-4fbf-93d2-07fd3f882876-kube-api-access-96x6j\") pod \"authorino-c548b5d7d-789f7\" (UID: \"3675fec3-3794-4fbf-93d2-07fd3f882876\") " pod="kuadrant-system/authorino-c548b5d7d-789f7" Sep 08 14:25:13.097643 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:13.097628 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-5fcsb" Sep 08 14:25:13.230592 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:13.230544 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-789f7" Sep 08 14:25:13.365430 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:13.365399 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-5fcsb"] Sep 08 14:25:13.366669 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:25:13.366643 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2218070e_23a8_49f1_bfe7_d177675b7e7c.slice/crio-daa8988b6f7d694f0823e71cb89c97cbbf9c50ccc1e1573aba56f3670123f0d1 WatchSource:0}: Error finding container daa8988b6f7d694f0823e71cb89c97cbbf9c50ccc1e1573aba56f3670123f0d1: Status 404 returned error can't find the container with id daa8988b6f7d694f0823e71cb89c97cbbf9c50ccc1e1573aba56f3670123f0d1 Sep 08 14:25:13.500779 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:13.500748 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-789f7"] Sep 08 14:25:13.503402 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:25:13.503372 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3675fec3_3794_4fbf_93d2_07fd3f882876.slice/crio-da2be6b3c970111b099cb46fd14b0d36012e93617ab1af9d7d8a53ba37dc3e18 WatchSource:0}: Error finding container da2be6b3c970111b099cb46fd14b0d36012e93617ab1af9d7d8a53ba37dc3e18: Status 404 returned error can't find the container with id da2be6b3c970111b099cb46fd14b0d36012e93617ab1af9d7d8a53ba37dc3e18 Sep 08 14:25:13.954422 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:13.954382 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" event={"ID":"79c20f6b-8b14-4b45-baae-d25702439256","Type":"ContainerStarted","Data":"3f7852439e9849c72d4edebc47535bb99502758878659fb46a3580135e56e745"} Sep 08 14:25:13.954655 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:13.954513 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" Sep 08 14:25:13.955688 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:13.955666 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-5fcsb" event={"ID":"2218070e-23a8-49f1-bfe7-d177675b7e7c","Type":"ContainerStarted","Data":"daa8988b6f7d694f0823e71cb89c97cbbf9c50ccc1e1573aba56f3670123f0d1"} Sep 08 14:25:13.956781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:13.956751 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-789f7" event={"ID":"3675fec3-3794-4fbf-93d2-07fd3f882876","Type":"ContainerStarted","Data":"da2be6b3c970111b099cb46fd14b0d36012e93617ab1af9d7d8a53ba37dc3e18"} Sep 08 14:25:13.971683 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:13.971643 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" podStartSLOduration=1.723522523 podStartE2EDuration="2.971628731s" totalImagesPullingTime="1.248106208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:25:13.969463835 +0000 UTC m=+526.253780976" watchObservedRunningTime="2026-09-08 14:25:13.971628731 +0000 UTC m=+526.255945900" Sep 08 14:25:15.973540 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:15.973503 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-5fcsb" event={"ID":"2218070e-23a8-49f1-bfe7-d177675b7e7c","Type":"ContainerStarted","Data":"8006f9c44bfe4c2787c014077ba5393aff0f0bfe9cc46520f8a8027f92522cf0"} Sep 08 14:25:15.975165 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:15.975136 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-789f7" event={"ID":"3675fec3-3794-4fbf-93d2-07fd3f882876","Type":"ContainerStarted","Data":"b00c8d7afc1967c09902a37ab7d7829c60249781eadc8eb02ff5e1b8204c4d78"} Sep 08 14:25:15.988872 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:15.988831 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-5fcsb" podStartSLOduration=1.811436695 podStartE2EDuration="3.988816466s" totalImagesPullingTime="2.177379771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:25:15.987804975 +0000 UTC m=+528.272122091" watchObservedRunningTime="2026-09-08 14:25:15.988816466 +0000 UTC m=+528.273133581" Sep 08 14:25:16.002095 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:16.002055 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-789f7" podStartSLOduration=1.9496838269999999 podStartE2EDuration="4.002041883s" totalImagesPullingTime="2.052358056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:25:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:25:16.001102854 +0000 UTC m=+528.285419982" watchObservedRunningTime="2026-09-08 14:25:16.002041883 +0000 UTC m=+528.286358998" Sep 08 14:25:16.026579 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:16.026552 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-5fcsb"] Sep 08 14:25:17.991592 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:17.991536 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-5fcsb" podUID="2218070e-23a8-49f1-bfe7-d177675b7e7c" containerName="authorino" containerID="cri-o://8006f9c44bfe4c2787c014077ba5393aff0f0bfe9cc46520f8a8027f92522cf0" gracePeriod=30 Sep 08 14:25:18.233048 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:18.233026 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-5fcsb" Sep 08 14:25:18.331406 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:18.331320 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-547hz\" (UniqueName: \"kubernetes.io/projected/2218070e-23a8-49f1-bfe7-d177675b7e7c-kube-api-access-547hz\") pod \"2218070e-23a8-49f1-bfe7-d177675b7e7c\" (UID: \"2218070e-23a8-49f1-bfe7-d177675b7e7c\") " Sep 08 14:25:18.333542 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:18.333517 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2218070e-23a8-49f1-bfe7-d177675b7e7c-kube-api-access-547hz" (OuterVolumeSpecName: "kube-api-access-547hz") pod "2218070e-23a8-49f1-bfe7-d177675b7e7c" (UID: "2218070e-23a8-49f1-bfe7-d177675b7e7c"). InnerVolumeSpecName "kube-api-access-547hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:25:18.432681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:18.432654 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-547hz\" (UniqueName: \"kubernetes.io/projected/2218070e-23a8-49f1-bfe7-d177675b7e7c-kube-api-access-547hz\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:25:18.997706 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:18.997673 2578 generic.go:356] "Generic (PLEG): container finished" podID="2218070e-23a8-49f1-bfe7-d177675b7e7c" containerID="8006f9c44bfe4c2787c014077ba5393aff0f0bfe9cc46520f8a8027f92522cf0" exitCode=0 Sep 08 14:25:18.998134 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:18.997721 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-5fcsb" Sep 08 14:25:18.998134 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:18.997766 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-5fcsb" event={"ID":"2218070e-23a8-49f1-bfe7-d177675b7e7c","Type":"ContainerDied","Data":"8006f9c44bfe4c2787c014077ba5393aff0f0bfe9cc46520f8a8027f92522cf0"} Sep 08 14:25:18.998134 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:18.997811 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-5fcsb" event={"ID":"2218070e-23a8-49f1-bfe7-d177675b7e7c","Type":"ContainerDied","Data":"daa8988b6f7d694f0823e71cb89c97cbbf9c50ccc1e1573aba56f3670123f0d1"} Sep 08 14:25:18.998134 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:18.997833 2578 scope.go:117] "RemoveContainer" containerID="8006f9c44bfe4c2787c014077ba5393aff0f0bfe9cc46520f8a8027f92522cf0" Sep 08 14:25:19.007373 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:19.007356 2578 scope.go:117] "RemoveContainer" containerID="8006f9c44bfe4c2787c014077ba5393aff0f0bfe9cc46520f8a8027f92522cf0" Sep 08 14:25:19.007665 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:25:19.007648 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8006f9c44bfe4c2787c014077ba5393aff0f0bfe9cc46520f8a8027f92522cf0\": container with ID starting with 8006f9c44bfe4c2787c014077ba5393aff0f0bfe9cc46520f8a8027f92522cf0 not found: ID does not exist" containerID="8006f9c44bfe4c2787c014077ba5393aff0f0bfe9cc46520f8a8027f92522cf0" Sep 08 14:25:19.007717 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:19.007674 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8006f9c44bfe4c2787c014077ba5393aff0f0bfe9cc46520f8a8027f92522cf0"} err="failed to get container status \"8006f9c44bfe4c2787c014077ba5393aff0f0bfe9cc46520f8a8027f92522cf0\": rpc error: code = NotFound desc = could not find container \"8006f9c44bfe4c2787c014077ba5393aff0f0bfe9cc46520f8a8027f92522cf0\": container with ID starting with 8006f9c44bfe4c2787c014077ba5393aff0f0bfe9cc46520f8a8027f92522cf0 not found: ID does not exist" Sep 08 14:25:19.018634 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:19.018613 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-5fcsb"] Sep 08 14:25:19.024391 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:19.024371 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-5fcsb"] Sep 08 14:25:20.242141 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:20.242105 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2218070e-23a8-49f1-bfe7-d177675b7e7c" path="/var/lib/kubelet/pods/2218070e-23a8-49f1-bfe7-d177675b7e7c/volumes" Sep 08 14:25:24.968371 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:24.968342 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" Sep 08 14:25:26.591462 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:26.591424 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pr89q"] Sep 08 14:25:26.591927 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:26.591648 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" podUID="79c20f6b-8b14-4b45-baae-d25702439256" containerName="limitador" containerID="cri-o://3f7852439e9849c72d4edebc47535bb99502758878659fb46a3580135e56e745" gracePeriod=30 Sep 08 14:25:27.042131 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:27.042099 2578 generic.go:356] "Generic (PLEG): container finished" podID="79c20f6b-8b14-4b45-baae-d25702439256" containerID="3f7852439e9849c72d4edebc47535bb99502758878659fb46a3580135e56e745" exitCode=0 Sep 08 14:25:27.042280 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:27.042177 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" event={"ID":"79c20f6b-8b14-4b45-baae-d25702439256","Type":"ContainerDied","Data":"3f7852439e9849c72d4edebc47535bb99502758878659fb46a3580135e56e745"} Sep 08 14:25:27.151963 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:27.151908 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" Sep 08 14:25:27.317552 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:27.317519 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzh7r\" (UniqueName: \"kubernetes.io/projected/79c20f6b-8b14-4b45-baae-d25702439256-kube-api-access-gzh7r\") pod \"79c20f6b-8b14-4b45-baae-d25702439256\" (UID: \"79c20f6b-8b14-4b45-baae-d25702439256\") " Sep 08 14:25:27.317696 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:27.317636 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79c20f6b-8b14-4b45-baae-d25702439256-config-file\") pod \"79c20f6b-8b14-4b45-baae-d25702439256\" (UID: \"79c20f6b-8b14-4b45-baae-d25702439256\") " Sep 08 14:25:27.318037 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:27.318015 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c20f6b-8b14-4b45-baae-d25702439256-config-file" (OuterVolumeSpecName: "config-file") pod "79c20f6b-8b14-4b45-baae-d25702439256" (UID: "79c20f6b-8b14-4b45-baae-d25702439256"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:25:27.319791 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:27.319771 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c20f6b-8b14-4b45-baae-d25702439256-kube-api-access-gzh7r" (OuterVolumeSpecName: "kube-api-access-gzh7r") pod "79c20f6b-8b14-4b45-baae-d25702439256" (UID: "79c20f6b-8b14-4b45-baae-d25702439256"). InnerVolumeSpecName "kube-api-access-gzh7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:25:27.418539 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:27.418450 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzh7r\" (UniqueName: \"kubernetes.io/projected/79c20f6b-8b14-4b45-baae-d25702439256-kube-api-access-gzh7r\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:25:27.418539 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:27.418487 2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/79c20f6b-8b14-4b45-baae-d25702439256-config-file\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:25:28.048348 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:28.048314 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" Sep 08 14:25:28.048348 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:28.048326 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-pr89q" event={"ID":"79c20f6b-8b14-4b45-baae-d25702439256","Type":"ContainerDied","Data":"998dae79b4a4077dc85cd1bf8bf99f8ebc73cd04b6bd9d3b6e2a1ae22cad209e"} Sep 08 14:25:28.048865 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:28.048387 2578 scope.go:117] "RemoveContainer" containerID="3f7852439e9849c72d4edebc47535bb99502758878659fb46a3580135e56e745" Sep 08 14:25:28.069299 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:28.069268 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pr89q"] Sep 08 14:25:28.073512 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:28.073491 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-pr89q"] Sep 08 14:25:28.241766 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:28.241722 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79c20f6b-8b14-4b45-baae-d25702439256" path="/var/lib/kubelet/pods/79c20f6b-8b14-4b45-baae-d25702439256/volumes" Sep 08 14:25:31.898797 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:31.898763 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk"] Sep 08 14:25:31.899280 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:31.899145 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2218070e-23a8-49f1-bfe7-d177675b7e7c" containerName="authorino" Sep 08 14:25:31.899280 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:31.899162 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2218070e-23a8-49f1-bfe7-d177675b7e7c" containerName="authorino" Sep 08 14:25:31.899280 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:31.899186 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c20f6b-8b14-4b45-baae-d25702439256" containerName="limitador" Sep 08 14:25:31.899280 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:31.899191 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c20f6b-8b14-4b45-baae-d25702439256" containerName="limitador" Sep 08 14:25:31.899280 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:31.899250 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c20f6b-8b14-4b45-baae-d25702439256" containerName="limitador" Sep 08 14:25:31.899280 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:31.899264 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2218070e-23a8-49f1-bfe7-d177675b7e7c" containerName="authorino" Sep 08 14:25:31.903618 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:31.903599 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk" Sep 08 14:25:31.906044 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:31.906023 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 08 14:25:31.906698 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:31.906682 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 08 14:25:31.906698 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:31.906694 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Sep 08 14:25:31.906880 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:31.906702 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tm5rw\"" Sep 08 14:25:31.912294 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:31.912274 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk"] Sep 08 14:25:32.058261 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:32.058232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jvht\" (UniqueName: \"kubernetes.io/projected/85ae4a8f-59ef-4909-820c-20c177c7513f-kube-api-access-6jvht\") pod \"postgres-69fb4594cf-2l4fk\" (UID: \"85ae4a8f-59ef-4909-820c-20c177c7513f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk" Sep 08 14:25:32.058261 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:32.058267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/85ae4a8f-59ef-4909-820c-20c177c7513f-data\") pod \"postgres-69fb4594cf-2l4fk\" (UID: \"85ae4a8f-59ef-4909-820c-20c177c7513f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk" Sep 08 14:25:32.159294 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:32.159204 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jvht\" (UniqueName: \"kubernetes.io/projected/85ae4a8f-59ef-4909-820c-20c177c7513f-kube-api-access-6jvht\") pod \"postgres-69fb4594cf-2l4fk\" (UID: \"85ae4a8f-59ef-4909-820c-20c177c7513f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk" Sep 08 14:25:32.159294 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:32.159244 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/85ae4a8f-59ef-4909-820c-20c177c7513f-data\") pod \"postgres-69fb4594cf-2l4fk\" (UID: \"85ae4a8f-59ef-4909-820c-20c177c7513f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk" Sep 08 14:25:32.159672 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:32.159650 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/85ae4a8f-59ef-4909-820c-20c177c7513f-data\") pod \"postgres-69fb4594cf-2l4fk\" (UID: \"85ae4a8f-59ef-4909-820c-20c177c7513f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk" Sep 08 14:25:32.168708 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:32.168679 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jvht\" (UniqueName: \"kubernetes.io/projected/85ae4a8f-59ef-4909-820c-20c177c7513f-kube-api-access-6jvht\") pod \"postgres-69fb4594cf-2l4fk\" (UID: \"85ae4a8f-59ef-4909-820c-20c177c7513f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk" Sep 08 14:25:32.215568 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:32.215548 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk" Sep 08 14:25:32.486389 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:32.486361 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk"] Sep 08 14:25:32.488071 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:25:32.488038 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ae4a8f_59ef_4909_820c_20c177c7513f.slice/crio-ff1449b5e391ac20d8216cd5974e849837a1bb10a838dea8c1c55bda77fadf74 WatchSource:0}: Error finding container ff1449b5e391ac20d8216cd5974e849837a1bb10a838dea8c1c55bda77fadf74: Status 404 returned error can't find the container with id ff1449b5e391ac20d8216cd5974e849837a1bb10a838dea8c1c55bda77fadf74 Sep 08 14:25:33.080125 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:33.080069 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk" event={"ID":"85ae4a8f-59ef-4909-820c-20c177c7513f","Type":"ContainerStarted","Data":"ff1449b5e391ac20d8216cd5974e849837a1bb10a838dea8c1c55bda77fadf74"} Sep 08 14:25:38.112294 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:38.112258 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk" event={"ID":"85ae4a8f-59ef-4909-820c-20c177c7513f","Type":"ContainerStarted","Data":"955a6bc5a982e338eadf5c600fd6aec802eb245e33b746b9e598df9f73eaa53a"} Sep 08 14:25:38.112796 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:38.112336 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk" Sep 08 14:25:38.128793 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:38.128716 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk" podStartSLOduration=2.461925937 podStartE2EDuration="7.128702303s" totalImagesPullingTime="4.666776366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:25:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:25:38.12797424 +0000 UTC m=+550.412291355" watchObservedRunningTime="2026-09-08 14:25:38.128702303 +0000 UTC m=+550.413019451" Sep 08 14:25:44.146638 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:44.146605 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-2l4fk" Sep 08 14:25:44.697060 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:44.697028 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-kdsqn"] Sep 08 14:25:44.700462 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:44.700447 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-kdsqn" Sep 08 14:25:44.707783 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:44.707757 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-kdsqn"] Sep 08 14:25:44.772507 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:44.772479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9gt7\" (UniqueName: \"kubernetes.io/projected/1838ed81-4de2-4a3d-98d6-71bde4c1b311-kube-api-access-b9gt7\") pod \"authorino-5dc7ccb5f-kdsqn\" (UID: \"1838ed81-4de2-4a3d-98d6-71bde4c1b311\") " pod="kuadrant-system/authorino-5dc7ccb5f-kdsqn" Sep 08 14:25:44.873493 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:44.873466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9gt7\" (UniqueName: \"kubernetes.io/projected/1838ed81-4de2-4a3d-98d6-71bde4c1b311-kube-api-access-b9gt7\") pod \"authorino-5dc7ccb5f-kdsqn\" (UID: \"1838ed81-4de2-4a3d-98d6-71bde4c1b311\") " pod="kuadrant-system/authorino-5dc7ccb5f-kdsqn" Sep 08 14:25:44.882021 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:44.881993 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9gt7\" (UniqueName: \"kubernetes.io/projected/1838ed81-4de2-4a3d-98d6-71bde4c1b311-kube-api-access-b9gt7\") pod \"authorino-5dc7ccb5f-kdsqn\" (UID: \"1838ed81-4de2-4a3d-98d6-71bde4c1b311\") " pod="kuadrant-system/authorino-5dc7ccb5f-kdsqn" Sep 08 14:25:44.944158 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:44.944130 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-kdsqn"] Sep 08 14:25:44.944426 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:44.944413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-kdsqn" Sep 08 14:25:44.968472 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:44.968447 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c89d688d5-9p8t5"] Sep 08 14:25:44.972909 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:44.972890 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c89d688d5-9p8t5" Sep 08 14:25:44.979029 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:44.978881 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c89d688d5-9p8t5"] Sep 08 14:25:45.073937 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.073901 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c89d688d5-9p8t5"] Sep 08 14:25:45.074234 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:25:45.074213 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-bjrhb], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6c89d688d5-9p8t5" podUID="435d4bcb-3adf-49da-a3a0-ceb5d8c6a649" Sep 08 14:25:45.074540 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.074523 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjrhb\" (UniqueName: \"kubernetes.io/projected/435d4bcb-3adf-49da-a3a0-ceb5d8c6a649-kube-api-access-bjrhb\") pod \"authorino-6c89d688d5-9p8t5\" (UID: \"435d4bcb-3adf-49da-a3a0-ceb5d8c6a649\") " pod="kuadrant-system/authorino-6c89d688d5-9p8t5" Sep 08 14:25:45.102205 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.102167 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-666cfb8df9-5fngs"] Sep 08 14:25:45.105585 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.105560 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-666cfb8df9-5fngs" Sep 08 14:25:45.107572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.107556 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 08 14:25:45.112130 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.112109 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-666cfb8df9-5fngs"] Sep 08 14:25:45.148291 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.148263 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c89d688d5-9p8t5" Sep 08 14:25:45.154794 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.154771 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c89d688d5-9p8t5" Sep 08 14:25:45.175193 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.175170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjrhb\" (UniqueName: \"kubernetes.io/projected/435d4bcb-3adf-49da-a3a0-ceb5d8c6a649-kube-api-access-bjrhb\") pod \"authorino-6c89d688d5-9p8t5\" (UID: \"435d4bcb-3adf-49da-a3a0-ceb5d8c6a649\") " pod="kuadrant-system/authorino-6c89d688d5-9p8t5" Sep 08 14:25:45.175343 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.175226 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03d8c75d-c7e0-4915-9209-6cc7af4bf172-tls-cert\") pod \"authorino-666cfb8df9-5fngs\" (UID: \"03d8c75d-c7e0-4915-9209-6cc7af4bf172\") " pod="kuadrant-system/authorino-666cfb8df9-5fngs" Sep 08 14:25:45.175343 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.175266 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rwkb\" (UniqueName: \"kubernetes.io/projected/03d8c75d-c7e0-4915-9209-6cc7af4bf172-kube-api-access-8rwkb\") pod \"authorino-666cfb8df9-5fngs\" (UID: \"03d8c75d-c7e0-4915-9209-6cc7af4bf172\") " pod="kuadrant-system/authorino-666cfb8df9-5fngs" Sep 08 14:25:45.185833 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.185804 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjrhb\" (UniqueName: \"kubernetes.io/projected/435d4bcb-3adf-49da-a3a0-ceb5d8c6a649-kube-api-access-bjrhb\") pod \"authorino-6c89d688d5-9p8t5\" (UID: \"435d4bcb-3adf-49da-a3a0-ceb5d8c6a649\") " pod="kuadrant-system/authorino-6c89d688d5-9p8t5" Sep 08 14:25:45.224206 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.224184 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-kdsqn"] Sep 08 14:25:45.225540 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:25:45.225506 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1838ed81_4de2_4a3d_98d6_71bde4c1b311.slice/crio-4c129197be3657dd16e59ebcd59699e02c2831f799686d3671c7de70d2d556ee WatchSource:0}: Error finding container 4c129197be3657dd16e59ebcd59699e02c2831f799686d3671c7de70d2d556ee: Status 404 returned error can't find the container with id 4c129197be3657dd16e59ebcd59699e02c2831f799686d3671c7de70d2d556ee Sep 08 14:25:45.275983 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.275958 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjrhb\" (UniqueName: \"kubernetes.io/projected/435d4bcb-3adf-49da-a3a0-ceb5d8c6a649-kube-api-access-bjrhb\") pod \"435d4bcb-3adf-49da-a3a0-ceb5d8c6a649\" (UID: \"435d4bcb-3adf-49da-a3a0-ceb5d8c6a649\") " Sep 08 14:25:45.276103 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.276090 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03d8c75d-c7e0-4915-9209-6cc7af4bf172-tls-cert\") pod \"authorino-666cfb8df9-5fngs\" (UID: \"03d8c75d-c7e0-4915-9209-6cc7af4bf172\") " pod="kuadrant-system/authorino-666cfb8df9-5fngs" Sep 08 14:25:45.276189 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.276174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rwkb\" (UniqueName: \"kubernetes.io/projected/03d8c75d-c7e0-4915-9209-6cc7af4bf172-kube-api-access-8rwkb\") pod \"authorino-666cfb8df9-5fngs\" (UID: \"03d8c75d-c7e0-4915-9209-6cc7af4bf172\") " pod="kuadrant-system/authorino-666cfb8df9-5fngs" Sep 08 14:25:45.278201 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.278180 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435d4bcb-3adf-49da-a3a0-ceb5d8c6a649-kube-api-access-bjrhb" (OuterVolumeSpecName: "kube-api-access-bjrhb") pod "435d4bcb-3adf-49da-a3a0-ceb5d8c6a649" (UID: "435d4bcb-3adf-49da-a3a0-ceb5d8c6a649"). InnerVolumeSpecName "kube-api-access-bjrhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:25:45.278638 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.278620 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03d8c75d-c7e0-4915-9209-6cc7af4bf172-tls-cert\") pod \"authorino-666cfb8df9-5fngs\" (UID: \"03d8c75d-c7e0-4915-9209-6cc7af4bf172\") " pod="kuadrant-system/authorino-666cfb8df9-5fngs" Sep 08 14:25:45.283787 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.283772 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rwkb\" (UniqueName: \"kubernetes.io/projected/03d8c75d-c7e0-4915-9209-6cc7af4bf172-kube-api-access-8rwkb\") pod \"authorino-666cfb8df9-5fngs\" (UID: \"03d8c75d-c7e0-4915-9209-6cc7af4bf172\") " pod="kuadrant-system/authorino-666cfb8df9-5fngs" Sep 08 14:25:45.377104 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.377076 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjrhb\" (UniqueName: \"kubernetes.io/projected/435d4bcb-3adf-49da-a3a0-ceb5d8c6a649-kube-api-access-bjrhb\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:25:45.414945 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.414911 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-666cfb8df9-5fngs" Sep 08 14:25:45.687988 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:45.687961 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-666cfb8df9-5fngs"] Sep 08 14:25:45.689401 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:25:45.689371 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03d8c75d_c7e0_4915_9209_6cc7af4bf172.slice/crio-6f2bfe10436d8186e08bf07c8b58a7b839c972597699b999faad0710ad7c7946 WatchSource:0}: Error finding container 6f2bfe10436d8186e08bf07c8b58a7b839c972597699b999faad0710ad7c7946: Status 404 returned error can't find the container with id 6f2bfe10436d8186e08bf07c8b58a7b839c972597699b999faad0710ad7c7946 Sep 08 14:25:46.161894 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:46.161854 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-666cfb8df9-5fngs" event={"ID":"03d8c75d-c7e0-4915-9209-6cc7af4bf172","Type":"ContainerStarted","Data":"6f2bfe10436d8186e08bf07c8b58a7b839c972597699b999faad0710ad7c7946"} Sep 08 14:25:46.163124 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:46.163098 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-kdsqn" event={"ID":"1838ed81-4de2-4a3d-98d6-71bde4c1b311","Type":"ContainerStarted","Data":"4c129197be3657dd16e59ebcd59699e02c2831f799686d3671c7de70d2d556ee"} Sep 08 14:25:46.163253 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:46.163156 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c89d688d5-9p8t5" Sep 08 14:25:46.192926 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:46.192899 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c89d688d5-9p8t5"] Sep 08 14:25:46.196490 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:46.196454 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c89d688d5-9p8t5"] Sep 08 14:25:46.241685 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:46.241658 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="435d4bcb-3adf-49da-a3a0-ceb5d8c6a649" path="/var/lib/kubelet/pods/435d4bcb-3adf-49da-a3a0-ceb5d8c6a649/volumes" Sep 08 14:25:47.170119 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:47.170080 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-kdsqn" event={"ID":"1838ed81-4de2-4a3d-98d6-71bde4c1b311","Type":"ContainerStarted","Data":"636dab21f227f2d6e0217f7b3456a29b985fa4f3eac9360a7f2a4cbe435df5f8"} Sep 08 14:25:47.170562 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:47.170164 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5dc7ccb5f-kdsqn" podUID="1838ed81-4de2-4a3d-98d6-71bde4c1b311" containerName="authorino" containerID="cri-o://636dab21f227f2d6e0217f7b3456a29b985fa4f3eac9360a7f2a4cbe435df5f8" gracePeriod=30 Sep 08 14:25:47.186799 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:47.186759 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dc7ccb5f-kdsqn" podStartSLOduration=2.244960422 podStartE2EDuration="3.186743775s" totalImagesPullingTime="941.783353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:25:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:25:47.184208068 +0000 UTC m=+559.468525183" watchObservedRunningTime="2026-09-08 14:25:47.186743775 +0000 UTC m=+559.471060880" Sep 08 14:25:47.419695 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:47.419674 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-kdsqn" Sep 08 14:25:47.495906 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:47.495876 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9gt7\" (UniqueName: \"kubernetes.io/projected/1838ed81-4de2-4a3d-98d6-71bde4c1b311-kube-api-access-b9gt7\") pod \"1838ed81-4de2-4a3d-98d6-71bde4c1b311\" (UID: \"1838ed81-4de2-4a3d-98d6-71bde4c1b311\") " Sep 08 14:25:47.498033 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:47.498009 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1838ed81-4de2-4a3d-98d6-71bde4c1b311-kube-api-access-b9gt7" (OuterVolumeSpecName: "kube-api-access-b9gt7") pod "1838ed81-4de2-4a3d-98d6-71bde4c1b311" (UID: "1838ed81-4de2-4a3d-98d6-71bde4c1b311"). InnerVolumeSpecName "kube-api-access-b9gt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:25:47.597027 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:47.596993 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9gt7\" (UniqueName: \"kubernetes.io/projected/1838ed81-4de2-4a3d-98d6-71bde4c1b311-kube-api-access-b9gt7\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:25:48.176296 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:48.176263 2578 generic.go:356] "Generic (PLEG): container finished" podID="1838ed81-4de2-4a3d-98d6-71bde4c1b311" containerID="636dab21f227f2d6e0217f7b3456a29b985fa4f3eac9360a7f2a4cbe435df5f8" exitCode=0 Sep 08 14:25:48.176787 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:48.176301 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-kdsqn" Sep 08 14:25:48.176787 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:48.176350 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-kdsqn" event={"ID":"1838ed81-4de2-4a3d-98d6-71bde4c1b311","Type":"ContainerDied","Data":"636dab21f227f2d6e0217f7b3456a29b985fa4f3eac9360a7f2a4cbe435df5f8"} Sep 08 14:25:48.176787 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:48.176400 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-kdsqn" event={"ID":"1838ed81-4de2-4a3d-98d6-71bde4c1b311","Type":"ContainerDied","Data":"4c129197be3657dd16e59ebcd59699e02c2831f799686d3671c7de70d2d556ee"} Sep 08 14:25:48.176787 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:48.176426 2578 scope.go:117] "RemoveContainer" containerID="636dab21f227f2d6e0217f7b3456a29b985fa4f3eac9360a7f2a4cbe435df5f8" Sep 08 14:25:48.185756 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:48.185720 2578 scope.go:117] "RemoveContainer" containerID="636dab21f227f2d6e0217f7b3456a29b985fa4f3eac9360a7f2a4cbe435df5f8" Sep 08 14:25:48.186020 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:25:48.186003 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636dab21f227f2d6e0217f7b3456a29b985fa4f3eac9360a7f2a4cbe435df5f8\": container with ID starting with 636dab21f227f2d6e0217f7b3456a29b985fa4f3eac9360a7f2a4cbe435df5f8 not found: ID does not exist" containerID="636dab21f227f2d6e0217f7b3456a29b985fa4f3eac9360a7f2a4cbe435df5f8" Sep 08 14:25:48.186088 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:48.186030 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636dab21f227f2d6e0217f7b3456a29b985fa4f3eac9360a7f2a4cbe435df5f8"} err="failed to get container status \"636dab21f227f2d6e0217f7b3456a29b985fa4f3eac9360a7f2a4cbe435df5f8\": rpc error: code = NotFound desc = could not find container \"636dab21f227f2d6e0217f7b3456a29b985fa4f3eac9360a7f2a4cbe435df5f8\": container with ID starting with 636dab21f227f2d6e0217f7b3456a29b985fa4f3eac9360a7f2a4cbe435df5f8 not found: ID does not exist" Sep 08 14:25:48.197411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:48.197385 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-kdsqn"] Sep 08 14:25:48.205066 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:48.205046 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-kdsqn"] Sep 08 14:25:48.241649 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:48.241626 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1838ed81-4de2-4a3d-98d6-71bde4c1b311" path="/var/lib/kubelet/pods/1838ed81-4de2-4a3d-98d6-71bde4c1b311/volumes" Sep 08 14:25:50.189110 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:50.189078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-666cfb8df9-5fngs" event={"ID":"03d8c75d-c7e0-4915-9209-6cc7af4bf172","Type":"ContainerStarted","Data":"a47178f7aacc03117f0ed40014e8c7b735ef259fe5f648bbf874b2dec152957a"} Sep 08 14:25:50.204275 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:50.204224 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-666cfb8df9-5fngs" podStartSLOduration=1.483333614 podStartE2EDuration="5.204206221s" totalImagesPullingTime="3.720872607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:25:50.20364739 +0000 UTC m=+562.487964510" watchObservedRunningTime="2026-09-08 14:25:50.204206221 +0000 UTC m=+562.488523335" Sep 08 14:25:50.229551 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:50.229525 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-789f7"] Sep 08 14:25:50.229764 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:50.229721 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-789f7" podUID="3675fec3-3794-4fbf-93d2-07fd3f882876" containerName="authorino" containerID="cri-o://b00c8d7afc1967c09902a37ab7d7829c60249781eadc8eb02ff5e1b8204c4d78" gracePeriod=30 Sep 08 14:25:50.477073 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:50.477043 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-789f7" Sep 08 14:25:50.522398 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:50.522357 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96x6j\" (UniqueName: \"kubernetes.io/projected/3675fec3-3794-4fbf-93d2-07fd3f882876-kube-api-access-96x6j\") pod \"3675fec3-3794-4fbf-93d2-07fd3f882876\" (UID: \"3675fec3-3794-4fbf-93d2-07fd3f882876\") " Sep 08 14:25:50.524700 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:50.524668 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3675fec3-3794-4fbf-93d2-07fd3f882876-kube-api-access-96x6j" (OuterVolumeSpecName: "kube-api-access-96x6j") pod "3675fec3-3794-4fbf-93d2-07fd3f882876" (UID: "3675fec3-3794-4fbf-93d2-07fd3f882876"). InnerVolumeSpecName "kube-api-access-96x6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:25:50.623791 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:50.623759 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96x6j\" (UniqueName: \"kubernetes.io/projected/3675fec3-3794-4fbf-93d2-07fd3f882876-kube-api-access-96x6j\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:25:51.195523 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:51.195491 2578 generic.go:356] "Generic (PLEG): container finished" podID="3675fec3-3794-4fbf-93d2-07fd3f882876" containerID="b00c8d7afc1967c09902a37ab7d7829c60249781eadc8eb02ff5e1b8204c4d78" exitCode=0 Sep 08 14:25:51.196240 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:51.195546 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-789f7" Sep 08 14:25:51.196240 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:51.195573 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-789f7" event={"ID":"3675fec3-3794-4fbf-93d2-07fd3f882876","Type":"ContainerDied","Data":"b00c8d7afc1967c09902a37ab7d7829c60249781eadc8eb02ff5e1b8204c4d78"} Sep 08 14:25:51.196240 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:51.195619 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-789f7" event={"ID":"3675fec3-3794-4fbf-93d2-07fd3f882876","Type":"ContainerDied","Data":"da2be6b3c970111b099cb46fd14b0d36012e93617ab1af9d7d8a53ba37dc3e18"} Sep 08 14:25:51.196240 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:51.195635 2578 scope.go:117] "RemoveContainer" containerID="b00c8d7afc1967c09902a37ab7d7829c60249781eadc8eb02ff5e1b8204c4d78" Sep 08 14:25:51.204805 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:51.204789 2578 scope.go:117] "RemoveContainer" containerID="b00c8d7afc1967c09902a37ab7d7829c60249781eadc8eb02ff5e1b8204c4d78" Sep 08 14:25:51.205068 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:25:51.205047 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00c8d7afc1967c09902a37ab7d7829c60249781eadc8eb02ff5e1b8204c4d78\": container with ID starting with b00c8d7afc1967c09902a37ab7d7829c60249781eadc8eb02ff5e1b8204c4d78 not found: ID does not exist" containerID="b00c8d7afc1967c09902a37ab7d7829c60249781eadc8eb02ff5e1b8204c4d78" Sep 08 14:25:51.205163 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:51.205074 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00c8d7afc1967c09902a37ab7d7829c60249781eadc8eb02ff5e1b8204c4d78"} err="failed to get container status \"b00c8d7afc1967c09902a37ab7d7829c60249781eadc8eb02ff5e1b8204c4d78\": rpc error: code = NotFound desc = could not find container \"b00c8d7afc1967c09902a37ab7d7829c60249781eadc8eb02ff5e1b8204c4d78\": container with ID starting with b00c8d7afc1967c09902a37ab7d7829c60249781eadc8eb02ff5e1b8204c4d78 not found: ID does not exist" Sep 08 14:25:51.216976 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:51.216955 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-789f7"] Sep 08 14:25:51.220962 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:51.220942 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-789f7"] Sep 08 14:25:52.241507 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:52.241473 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3675fec3-3794-4fbf-93d2-07fd3f882876" path="/var/lib/kubelet/pods/3675fec3-3794-4fbf-93d2-07fd3f882876/volumes" Sep 08 14:25:55.374435 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.374404 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-757f9dc676-vkhkz"] Sep 08 14:25:55.374807 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.374752 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1838ed81-4de2-4a3d-98d6-71bde4c1b311" containerName="authorino" Sep 08 14:25:55.374807 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.374763 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1838ed81-4de2-4a3d-98d6-71bde4c1b311" containerName="authorino" Sep 08 14:25:55.374807 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.374779 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3675fec3-3794-4fbf-93d2-07fd3f882876" containerName="authorino" Sep 08 14:25:55.374807 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.374785 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3675fec3-3794-4fbf-93d2-07fd3f882876" containerName="authorino" Sep 08 14:25:55.374984 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.374855 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3675fec3-3794-4fbf-93d2-07fd3f882876" containerName="authorino" Sep 08 14:25:55.374984 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.374865 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1838ed81-4de2-4a3d-98d6-71bde4c1b311" containerName="authorino" Sep 08 14:25:55.377813 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.377795 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:25:55.380146 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.380026 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 14:25:55.380146 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.380058 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 14:25:55.380146 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.380126 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-czc9x\"" Sep 08 14:25:55.380784 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.380754 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 14:25:55.386946 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.386924 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-757f9dc676-vkhkz"] Sep 08 14:25:55.463906 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.463868 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd5kj\" (UniqueName: \"kubernetes.io/projected/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-kube-api-access-cd5kj\") pod \"maas-controller-757f9dc676-vkhkz\" (UID: \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\") " pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:25:55.464075 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.463929 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-metrics-certs\") pod \"maas-controller-757f9dc676-vkhkz\" (UID: \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\") " pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:25:55.464075 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.464026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-webhook-cert\") pod \"maas-controller-757f9dc676-vkhkz\" (UID: \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\") " pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:25:55.541289 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.541260 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cb4b89648-78rbv"] Sep 08 14:25:55.543860 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.543842 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:25:55.556690 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.556661 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb4b89648-78rbv"] Sep 08 14:25:55.564840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.564816 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd5kj\" (UniqueName: \"kubernetes.io/projected/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-kube-api-access-cd5kj\") pod \"maas-controller-757f9dc676-vkhkz\" (UID: \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\") " pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:25:55.564931 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.564861 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-metrics-certs\") pod \"maas-controller-757f9dc676-vkhkz\" (UID: \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\") " pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:25:55.564931 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.564910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-webhook-cert\") pod \"maas-controller-757f9dc676-vkhkz\" (UID: \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\") " pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:25:55.565043 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:25:55.565029 2578 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Sep 08 14:25:55.565115 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:25:55.565105 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-webhook-cert podName:8a2ced64-ca0a-4d59-8e86-cb8305888e8a nodeName:}" failed. No retries permitted until 2026-09-08 14:25:56.065084526 +0000 UTC m=+568.349401620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-webhook-cert") pod "maas-controller-757f9dc676-vkhkz" (UID: "8a2ced64-ca0a-4d59-8e86-cb8305888e8a") : secret "maas-controller-webhook-cert" not found Sep 08 14:25:55.567421 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.567403 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-metrics-certs\") pod \"maas-controller-757f9dc676-vkhkz\" (UID: \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\") " pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:25:55.575225 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.575202 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd5kj\" (UniqueName: \"kubernetes.io/projected/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-kube-api-access-cd5kj\") pod \"maas-controller-757f9dc676-vkhkz\" (UID: \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\") " pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:25:55.665768 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.665668 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t962p\" (UniqueName: \"kubernetes.io/projected/e55c94b0-218e-4184-abd6-ad53f40b8b19-kube-api-access-t962p\") pod \"maas-controller-6cb4b89648-78rbv\" (UID: \"e55c94b0-218e-4184-abd6-ad53f40b8b19\") " pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:25:55.665768 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.665714 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e55c94b0-218e-4184-abd6-ad53f40b8b19-webhook-cert\") pod \"maas-controller-6cb4b89648-78rbv\" (UID: \"e55c94b0-218e-4184-abd6-ad53f40b8b19\") " pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:25:55.666016 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.665835 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55c94b0-218e-4184-abd6-ad53f40b8b19-metrics-certs\") pod \"maas-controller-6cb4b89648-78rbv\" (UID: \"e55c94b0-218e-4184-abd6-ad53f40b8b19\") " pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:25:55.766966 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.766934 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55c94b0-218e-4184-abd6-ad53f40b8b19-metrics-certs\") pod \"maas-controller-6cb4b89648-78rbv\" (UID: \"e55c94b0-218e-4184-abd6-ad53f40b8b19\") " pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:25:55.767126 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.767029 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t962p\" (UniqueName: \"kubernetes.io/projected/e55c94b0-218e-4184-abd6-ad53f40b8b19-kube-api-access-t962p\") pod \"maas-controller-6cb4b89648-78rbv\" (UID: \"e55c94b0-218e-4184-abd6-ad53f40b8b19\") " pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:25:55.767126 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.767077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e55c94b0-218e-4184-abd6-ad53f40b8b19-webhook-cert\") pod \"maas-controller-6cb4b89648-78rbv\" (UID: \"e55c94b0-218e-4184-abd6-ad53f40b8b19\") " pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:25:55.769575 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.769550 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55c94b0-218e-4184-abd6-ad53f40b8b19-metrics-certs\") pod \"maas-controller-6cb4b89648-78rbv\" (UID: \"e55c94b0-218e-4184-abd6-ad53f40b8b19\") " pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:25:55.769690 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.769605 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e55c94b0-218e-4184-abd6-ad53f40b8b19-webhook-cert\") pod \"maas-controller-6cb4b89648-78rbv\" (UID: \"e55c94b0-218e-4184-abd6-ad53f40b8b19\") " pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:25:55.774959 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.774939 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t962p\" (UniqueName: \"kubernetes.io/projected/e55c94b0-218e-4184-abd6-ad53f40b8b19-kube-api-access-t962p\") pod \"maas-controller-6cb4b89648-78rbv\" (UID: \"e55c94b0-218e-4184-abd6-ad53f40b8b19\") " pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:25:55.856076 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:55.856037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:25:56.070680 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:56.070650 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-webhook-cert\") pod \"maas-controller-757f9dc676-vkhkz\" (UID: \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\") " pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:25:56.073208 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:56.073189 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-webhook-cert\") pod \"maas-controller-757f9dc676-vkhkz\" (UID: \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\") " pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:25:56.127931 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:56.127907 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb4b89648-78rbv"] Sep 08 14:25:56.129523 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:25:56.129495 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode55c94b0_218e_4184_abd6_ad53f40b8b19.slice/crio-44e8cb12cb5ba3976dae5585b806740b25b70180450bbd223916180e5bd75a68 WatchSource:0}: Error finding container 44e8cb12cb5ba3976dae5585b806740b25b70180450bbd223916180e5bd75a68: Status 404 returned error can't find the container with id 44e8cb12cb5ba3976dae5585b806740b25b70180450bbd223916180e5bd75a68 Sep 08 14:25:56.223537 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:56.223503 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb4b89648-78rbv" event={"ID":"e55c94b0-218e-4184-abd6-ad53f40b8b19","Type":"ContainerStarted","Data":"44e8cb12cb5ba3976dae5585b806740b25b70180450bbd223916180e5bd75a68"} Sep 08 14:25:56.289382 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:56.289351 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:25:56.561650 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:56.561622 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-757f9dc676-vkhkz"] Sep 08 14:25:56.562711 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:25:56.562684 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a2ced64_ca0a_4d59_8e86_cb8305888e8a.slice/crio-1b685e39c60c781c1c0daf1b0d445779c19fc9a308db494e0c4043dca6a133bb WatchSource:0}: Error finding container 1b685e39c60c781c1c0daf1b0d445779c19fc9a308db494e0c4043dca6a133bb: Status 404 returned error can't find the container with id 1b685e39c60c781c1c0daf1b0d445779c19fc9a308db494e0c4043dca6a133bb Sep 08 14:25:57.234701 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:57.234624 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-757f9dc676-vkhkz" event={"ID":"8a2ced64-ca0a-4d59-8e86-cb8305888e8a","Type":"ContainerStarted","Data":"1b685e39c60c781c1c0daf1b0d445779c19fc9a308db494e0c4043dca6a133bb"} Sep 08 14:25:59.249404 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:59.249372 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb4b89648-78rbv" event={"ID":"e55c94b0-218e-4184-abd6-ad53f40b8b19","Type":"ContainerStarted","Data":"b2807e64c7584f1f52fe4a5006ec2bd5e98c8829774fd7454fd8eb1eb338b34b"} Sep 08 14:25:59.249854 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:59.249446 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:25:59.250926 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:59.250903 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-757f9dc676-vkhkz" event={"ID":"8a2ced64-ca0a-4d59-8e86-cb8305888e8a","Type":"ContainerStarted","Data":"15a9ca62585d7e2b839b947c6b5b12403a510a1bd4a1fe158c8ac64129bdcfb1"} Sep 08 14:25:59.251056 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:59.250954 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:25:59.270871 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:59.270825 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cb4b89648-78rbv" podStartSLOduration=1.625473594 podStartE2EDuration="4.270810234s" totalImagesPullingTime="2.64533664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:25:59.26755185 +0000 UTC m=+571.551868964" watchObservedRunningTime="2026-09-08 14:25:59.270810234 +0000 UTC m=+571.555127349" Sep 08 14:25:59.286239 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:25:59.286189 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-757f9dc676-vkhkz" podStartSLOduration=2.0675028109999998 podStartE2EDuration="4.286175583s" totalImagesPullingTime="2.218672772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:25:59.285108246 +0000 UTC m=+571.569425361" watchObservedRunningTime="2026-09-08 14:25:59.286175583 +0000 UTC m=+571.570492698" Sep 08 14:26:10.258589 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.258559 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:26:10.258979 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.258621 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:26:10.321144 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.321112 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-757f9dc676-vkhkz"] Sep 08 14:26:10.321350 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.321326 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-757f9dc676-vkhkz" podUID="8a2ced64-ca0a-4d59-8e86-cb8305888e8a" containerName="manager" containerID="cri-o://15a9ca62585d7e2b839b947c6b5b12403a510a1bd4a1fe158c8ac64129bdcfb1" gracePeriod=30 Sep 08 14:26:10.565392 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.565370 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:26:10.708750 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.708692 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-webhook-cert\") pod \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\" (UID: \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\") " Sep 08 14:26:10.708925 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.708836 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-metrics-certs\") pod \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\" (UID: \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\") " Sep 08 14:26:10.708925 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.708875 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cd5kj\" (UniqueName: \"kubernetes.io/projected/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-kube-api-access-cd5kj\") pod \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\" (UID: \"8a2ced64-ca0a-4d59-8e86-cb8305888e8a\") " Sep 08 14:26:10.711113 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.711073 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8a2ced64-ca0a-4d59-8e86-cb8305888e8a" (UID: "8a2ced64-ca0a-4d59-8e86-cb8305888e8a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:26:10.711113 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.711086 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8a2ced64-ca0a-4d59-8e86-cb8305888e8a" (UID: "8a2ced64-ca0a-4d59-8e86-cb8305888e8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:26:10.711260 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.711148 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-kube-api-access-cd5kj" (OuterVolumeSpecName: "kube-api-access-cd5kj") pod "8a2ced64-ca0a-4d59-8e86-cb8305888e8a" (UID: "8a2ced64-ca0a-4d59-8e86-cb8305888e8a"). InnerVolumeSpecName "kube-api-access-cd5kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:26:10.810060 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.809974 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:26:10.810060 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.810002 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd5kj\" (UniqueName: \"kubernetes.io/projected/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-kube-api-access-cd5kj\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:26:10.810060 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:10.810016 2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8a2ced64-ca0a-4d59-8e86-cb8305888e8a-webhook-cert\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:26:11.321270 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:11.321230 2578 generic.go:356] "Generic (PLEG): container finished" podID="8a2ced64-ca0a-4d59-8e86-cb8305888e8a" containerID="15a9ca62585d7e2b839b947c6b5b12403a510a1bd4a1fe158c8ac64129bdcfb1" exitCode=0 Sep 08 14:26:11.321667 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:11.321294 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-757f9dc676-vkhkz" Sep 08 14:26:11.321667 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:11.321308 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-757f9dc676-vkhkz" event={"ID":"8a2ced64-ca0a-4d59-8e86-cb8305888e8a","Type":"ContainerDied","Data":"15a9ca62585d7e2b839b947c6b5b12403a510a1bd4a1fe158c8ac64129bdcfb1"} Sep 08 14:26:11.321667 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:11.321383 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-757f9dc676-vkhkz" event={"ID":"8a2ced64-ca0a-4d59-8e86-cb8305888e8a","Type":"ContainerDied","Data":"1b685e39c60c781c1c0daf1b0d445779c19fc9a308db494e0c4043dca6a133bb"} Sep 08 14:26:11.321667 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:11.321404 2578 scope.go:117] "RemoveContainer" containerID="15a9ca62585d7e2b839b947c6b5b12403a510a1bd4a1fe158c8ac64129bdcfb1" Sep 08 14:26:11.330841 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:11.330822 2578 scope.go:117] "RemoveContainer" containerID="15a9ca62585d7e2b839b947c6b5b12403a510a1bd4a1fe158c8ac64129bdcfb1" Sep 08 14:26:11.331112 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:26:11.331092 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a9ca62585d7e2b839b947c6b5b12403a510a1bd4a1fe158c8ac64129bdcfb1\": container with ID starting with 15a9ca62585d7e2b839b947c6b5b12403a510a1bd4a1fe158c8ac64129bdcfb1 not found: ID does not exist" containerID="15a9ca62585d7e2b839b947c6b5b12403a510a1bd4a1fe158c8ac64129bdcfb1" Sep 08 14:26:11.331175 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:11.331125 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a9ca62585d7e2b839b947c6b5b12403a510a1bd4a1fe158c8ac64129bdcfb1"} err="failed to get container status \"15a9ca62585d7e2b839b947c6b5b12403a510a1bd4a1fe158c8ac64129bdcfb1\": rpc error: code = NotFound desc = could not find container \"15a9ca62585d7e2b839b947c6b5b12403a510a1bd4a1fe158c8ac64129bdcfb1\": container with ID starting with 15a9ca62585d7e2b839b947c6b5b12403a510a1bd4a1fe158c8ac64129bdcfb1 not found: ID does not exist" Sep 08 14:26:11.342260 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:11.342237 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-757f9dc676-vkhkz"] Sep 08 14:26:11.349114 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:11.349093 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-757f9dc676-vkhkz"] Sep 08 14:26:12.242001 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:12.241971 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a2ced64-ca0a-4d59-8e86-cb8305888e8a" path="/var/lib/kubelet/pods/8a2ced64-ca0a-4d59-8e86-cb8305888e8a/volumes" Sep 08 14:26:28.154861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:28.154827 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:26:28.155497 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:28.155478 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:26:28.162538 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:28.162516 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:26:28.163048 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:28.163027 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:26:33.222990 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.222949 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8554757b-b26bb"] Sep 08 14:26:33.227321 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.223460 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a2ced64-ca0a-4d59-8e86-cb8305888e8a" containerName="manager" Sep 08 14:26:33.227321 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.223480 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a2ced64-ca0a-4d59-8e86-cb8305888e8a" containerName="manager" Sep 08 14:26:33.227321 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.223580 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a2ced64-ca0a-4d59-8e86-cb8305888e8a" containerName="manager" Sep 08 14:26:33.227321 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.227312 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:33.230148 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.230124 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 08 14:26:33.230148 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.230134 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 08 14:26:33.230330 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.230135 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dck9z\"" Sep 08 14:26:33.237996 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.237974 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8554757b-b26bb"] Sep 08 14:26:33.274435 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.274405 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-metrics-certs\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:33.274580 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.274457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrbf\" (UniqueName: \"kubernetes.io/projected/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-kube-api-access-lrrbf\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:33.274580 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.274513 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-maas-api-tls\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:33.274580 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.274536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-openshift-service-ca\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:33.375524 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.375486 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-metrics-certs\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:33.375719 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.375534 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrbf\" (UniqueName: \"kubernetes.io/projected/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-kube-api-access-lrrbf\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:33.375719 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.375560 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-maas-api-tls\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:33.375719 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.375574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-openshift-service-ca\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:33.375719 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:26:33.375645 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found Sep 08 14:26:33.375719 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:26:33.375720 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-metrics-certs podName:1cd4800d-b7f7-43bc-a7de-f10389bd4ce9 nodeName:}" failed. No retries permitted until 2026-09-08 14:26:33.875699397 +0000 UTC m=+606.160016490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-metrics-certs") pod "maas-api-7f8554757b-b26bb" (UID: "1cd4800d-b7f7-43bc-a7de-f10389bd4ce9") : secret "maas-api-metrics-tls" not found Sep 08 14:26:33.376298 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.376280 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-openshift-service-ca\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:33.378339 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.378303 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-maas-api-tls\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:33.384842 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.384817 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrbf\" (UniqueName: \"kubernetes.io/projected/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-kube-api-access-lrrbf\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:33.754748 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.754702 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd"] Sep 08 14:26:33.757301 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.757286 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd" Sep 08 14:26:33.759542 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.759523 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 08 14:26:33.771661 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.771640 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd"] Sep 08 14:26:33.779070 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.779047 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5385bcf-56ea-4679-87b6-44a041f2a606-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-kzbbd\" (UID: \"a5385bcf-56ea-4679-87b6-44a041f2a606\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd" Sep 08 14:26:33.779162 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.779108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n2xm\" (UniqueName: \"kubernetes.io/projected/a5385bcf-56ea-4679-87b6-44a041f2a606-kube-api-access-2n2xm\") pod \"payload-pre-processing-59fbc47bf8-kzbbd\" (UID: \"a5385bcf-56ea-4679-87b6-44a041f2a606\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd" Sep 08 14:26:33.811096 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.811068 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-9xh8f"] Sep 08 14:26:33.813489 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.813474 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-9xh8f" Sep 08 14:26:33.815827 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.815803 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-bmctb\"" Sep 08 14:26:33.825276 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.824249 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-9xh8f"] Sep 08 14:26:33.880573 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.880530 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5385bcf-56ea-4679-87b6-44a041f2a606-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-kzbbd\" (UID: \"a5385bcf-56ea-4679-87b6-44a041f2a606\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd" Sep 08 14:26:33.880820 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.880660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n2xm\" (UniqueName: \"kubernetes.io/projected/a5385bcf-56ea-4679-87b6-44a041f2a606-kube-api-access-2n2xm\") pod \"payload-pre-processing-59fbc47bf8-kzbbd\" (UID: \"a5385bcf-56ea-4679-87b6-44a041f2a606\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd" Sep 08 14:26:33.880820 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.880705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88bb47cd-a680-4da8-8e34-092fb40d590d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-9xh8f\" (UID: \"88bb47cd-a680-4da8-8e34-092fb40d590d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9xh8f" Sep 08 14:26:33.880820 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.880783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnlj9\" (UniqueName: \"kubernetes.io/projected/88bb47cd-a680-4da8-8e34-092fb40d590d-kube-api-access-vnlj9\") pod \"payload-processing-5b54b7fbb7-9xh8f\" (UID: \"88bb47cd-a680-4da8-8e34-092fb40d590d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9xh8f" Sep 08 14:26:33.881086 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.880832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-metrics-certs\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:33.881086 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:26:33.880951 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found Sep 08 14:26:33.881086 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:26:33.881001 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-metrics-certs podName:1cd4800d-b7f7-43bc-a7de-f10389bd4ce9 nodeName:}" failed. No retries permitted until 2026-09-08 14:26:34.880987739 +0000 UTC m=+607.165304832 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-metrics-certs") pod "maas-api-7f8554757b-b26bb" (UID: "1cd4800d-b7f7-43bc-a7de-f10389bd4ce9") : secret "maas-api-metrics-tls" not found Sep 08 14:26:33.881467 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.881445 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5385bcf-56ea-4679-87b6-44a041f2a606-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-kzbbd\" (UID: \"a5385bcf-56ea-4679-87b6-44a041f2a606\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd" Sep 08 14:26:33.888811 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.888791 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n2xm\" (UniqueName: \"kubernetes.io/projected/a5385bcf-56ea-4679-87b6-44a041f2a606-kube-api-access-2n2xm\") pod \"payload-pre-processing-59fbc47bf8-kzbbd\" (UID: \"a5385bcf-56ea-4679-87b6-44a041f2a606\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd" Sep 08 14:26:33.981723 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.981682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88bb47cd-a680-4da8-8e34-092fb40d590d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-9xh8f\" (UID: \"88bb47cd-a680-4da8-8e34-092fb40d590d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9xh8f" Sep 08 14:26:33.981939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.981800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnlj9\" (UniqueName: \"kubernetes.io/projected/88bb47cd-a680-4da8-8e34-092fb40d590d-kube-api-access-vnlj9\") pod \"payload-processing-5b54b7fbb7-9xh8f\" (UID: \"88bb47cd-a680-4da8-8e34-092fb40d590d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9xh8f" Sep 08 14:26:33.982523 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.982499 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88bb47cd-a680-4da8-8e34-092fb40d590d-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-9xh8f\" (UID: \"88bb47cd-a680-4da8-8e34-092fb40d590d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9xh8f" Sep 08 14:26:33.991196 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:33.991167 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnlj9\" (UniqueName: \"kubernetes.io/projected/88bb47cd-a680-4da8-8e34-092fb40d590d-kube-api-access-vnlj9\") pod \"payload-processing-5b54b7fbb7-9xh8f\" (UID: \"88bb47cd-a680-4da8-8e34-092fb40d590d\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-9xh8f" Sep 08 14:26:34.068165 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:34.068077 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd" Sep 08 14:26:34.127756 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:34.127474 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-9xh8f" Sep 08 14:26:34.347882 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:34.347847 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd"] Sep 08 14:26:34.350096 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:26:34.350068 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5385bcf_56ea_4679_87b6_44a041f2a606.slice/crio-9855f58ee51aada0063a54b5e24a80d19bf99f147c39ce2b605edb890e607a82 WatchSource:0}: Error finding container 9855f58ee51aada0063a54b5e24a80d19bf99f147c39ce2b605edb890e607a82: Status 404 returned error can't find the container with id 9855f58ee51aada0063a54b5e24a80d19bf99f147c39ce2b605edb890e607a82 Sep 08 14:26:34.429923 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:34.429884 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-9xh8f"] Sep 08 14:26:34.430114 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:26:34.430092 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bb47cd_a680_4da8_8e34_092fb40d590d.slice/crio-1edcad53d496aec079ec32e0d75b6e4eb9e2626c429a8262f5c7089f71972e1c WatchSource:0}: Error finding container 1edcad53d496aec079ec32e0d75b6e4eb9e2626c429a8262f5c7089f71972e1c: Status 404 returned error can't find the container with id 1edcad53d496aec079ec32e0d75b6e4eb9e2626c429a8262f5c7089f71972e1c Sep 08 14:26:34.441947 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:34.441914 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-9xh8f" event={"ID":"88bb47cd-a680-4da8-8e34-092fb40d590d","Type":"ContainerStarted","Data":"1edcad53d496aec079ec32e0d75b6e4eb9e2626c429a8262f5c7089f71972e1c"} Sep 08 14:26:34.443236 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:34.443214 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd" event={"ID":"a5385bcf-56ea-4679-87b6-44a041f2a606","Type":"ContainerStarted","Data":"9855f58ee51aada0063a54b5e24a80d19bf99f147c39ce2b605edb890e607a82"} Sep 08 14:26:34.907945 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:34.907904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-metrics-certs\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:34.910811 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:34.910786 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-metrics-certs\") pod \"maas-api-7f8554757b-b26bb\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:35.038476 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:35.038396 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:35.352386 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:35.352349 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8554757b-b26bb"] Sep 08 14:26:35.354471 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:26:35.354430 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd4800d_b7f7_43bc_a7de_f10389bd4ce9.slice/crio-a513072b18806808a538ccd8181c06a5106f7bec596db26c12a37a8d152f4d62 WatchSource:0}: Error finding container a513072b18806808a538ccd8181c06a5106f7bec596db26c12a37a8d152f4d62: Status 404 returned error can't find the container with id a513072b18806808a538ccd8181c06a5106f7bec596db26c12a37a8d152f4d62 Sep 08 14:26:35.453198 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:35.453118 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" event={"ID":"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9","Type":"ContainerStarted","Data":"a513072b18806808a538ccd8181c06a5106f7bec596db26c12a37a8d152f4d62"} Sep 08 14:26:37.467339 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:37.467306 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd" event={"ID":"a5385bcf-56ea-4679-87b6-44a041f2a606","Type":"ContainerStarted","Data":"241dfea62e96facf7b575fa24f796ed738f06c85bcab2ee2fe7e3bc8189a309a"} Sep 08 14:26:37.467832 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:37.467388 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd" Sep 08 14:26:37.468958 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:37.468936 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-9xh8f" event={"ID":"88bb47cd-a680-4da8-8e34-092fb40d590d","Type":"ContainerStarted","Data":"0797d9bc1a74b0b1fb393d928c81e23a3f1d8b0ac6893dda73e0a79c4b46a48c"} Sep 08 14:26:37.469071 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:37.468992 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-9xh8f" Sep 08 14:26:37.485900 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:37.485848 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd" podStartSLOduration=2.073889726 podStartE2EDuration="4.485828698s" totalImagesPullingTime="2.411938972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:26:37.482597996 +0000 UTC m=+609.766915114" watchObservedRunningTime="2026-09-08 14:26:37.485828698 +0000 UTC m=+609.770145814" Sep 08 14:26:37.499991 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:37.499940 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-9xh8f" podStartSLOduration=2.174434459 podStartE2EDuration="4.499921096s" totalImagesPullingTime="2.325486637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:26:37.497613776 +0000 UTC m=+609.781930893" watchObservedRunningTime="2026-09-08 14:26:37.499921096 +0000 UTC m=+609.784238208" Sep 08 14:26:39.490407 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:39.489460 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" event={"ID":"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9","Type":"ContainerStarted","Data":"b650880cab08637054e3afe0514c9a4d66032b316516325065abf84e2af0e959"} Sep 08 14:26:39.490407 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:39.490345 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:39.507803 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:39.507745 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" podStartSLOduration=2.496714771 podStartE2EDuration="6.507707425s" totalImagesPullingTime="4.010992654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:26:39.506080696 +0000 UTC m=+611.790397813" watchObservedRunningTime="2026-09-08 14:26:39.507707425 +0000 UTC m=+611.792024543" Sep 08 14:26:43.516039 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:43.516007 2578 generic.go:356] "Generic (PLEG): container finished" podID="07ac3b36-1d40-4ebe-a52f-31237baa182e" containerID="39edd21109637bdff419f5e5dc15728612ef7a18d20788998d5b08c41adeda16" exitCode=0 Sep 08 14:26:43.516441 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:43.516082 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" event={"ID":"07ac3b36-1d40-4ebe-a52f-31237baa182e","Type":"ContainerDied","Data":"39edd21109637bdff419f5e5dc15728612ef7a18d20788998d5b08c41adeda16"} Sep 08 14:26:43.516485 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:43.516450 2578 scope.go:117] "RemoveContainer" containerID="39edd21109637bdff419f5e5dc15728612ef7a18d20788998d5b08c41adeda16" Sep 08 14:26:43.517442 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:43.517411 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 08 14:26:43.517524 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:43.517509 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 08 14:26:43.517568 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:43.517556 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 08 14:26:44.523639 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:44.523608 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" event={"ID":"07ac3b36-1d40-4ebe-a52f-31237baa182e","Type":"ContainerStarted","Data":"e20f2ba22830b60712f0d35b741be93ffb66a5794fc98747cd882eb9d890b32a"} Sep 08 14:26:44.869044 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:44.868948 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:26:45.447062 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:45.447029 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:26:45.448300 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:45.448274 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-jwqsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 08 14:26:45.448425 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:45.448335 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" podUID="07ac3b36-1d40-4ebe-a52f-31237baa182e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 08 14:26:46.446638 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:46.446597 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-jwqsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 08 14:26:46.447062 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:46.446668 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" podUID="07ac3b36-1d40-4ebe-a52f-31237baa182e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 08 14:26:46.508549 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:46.508520 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:26:47.446603 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:47.446560 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-jwqsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 08 14:26:47.446812 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:47.446631 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" podUID="07ac3b36-1d40-4ebe-a52f-31237baa182e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 08 14:26:48.447173 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:48.447130 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-jwqsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 08 14:26:48.447629 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:48.447192 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" podUID="07ac3b36-1d40-4ebe-a52f-31237baa182e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 08 14:26:48.478224 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:48.478192 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-9xh8f" Sep 08 14:26:48.478387 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:48.478270 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kzbbd" Sep 08 14:26:49.447471 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:49.447427 2578 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-jwqsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Sep 08 14:26:49.447889 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:49.447507 2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" podUID="07ac3b36-1d40-4ebe-a52f-31237baa182e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Sep 08 14:26:50.450158 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:50.450129 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:26:50.451019 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:26:50.451000 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-jwqsz" Sep 08 14:27:18.735185 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.735141 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626"] Sep 08 14:27:18.740676 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.740651 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.743840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.743812 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 08 14:27:18.744036 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.743958 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 08 14:27:18.744036 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.743958 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-68fbj\"" Sep 08 14:27:18.744149 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.744084 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 08 14:27:18.747284 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.747264 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626"] Sep 08 14:27:18.818304 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.818269 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/112c52fa-a621-4eca-adda-3aa16e25a4d2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.818481 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.818323 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hxfp\" (UniqueName: \"kubernetes.io/projected/112c52fa-a621-4eca-adda-3aa16e25a4d2-kube-api-access-5hxfp\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.818481 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.818417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112c52fa-a621-4eca-adda-3aa16e25a4d2-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.818481 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.818447 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/112c52fa-a621-4eca-adda-3aa16e25a4d2-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.818481 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.818481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/112c52fa-a621-4eca-adda-3aa16e25a4d2-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.818621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.818494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/112c52fa-a621-4eca-adda-3aa16e25a4d2-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.919424 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.919386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112c52fa-a621-4eca-adda-3aa16e25a4d2-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.919621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.919439 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/112c52fa-a621-4eca-adda-3aa16e25a4d2-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.919621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.919489 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/112c52fa-a621-4eca-adda-3aa16e25a4d2-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.919621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.919513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/112c52fa-a621-4eca-adda-3aa16e25a4d2-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.919621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.919583 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/112c52fa-a621-4eca-adda-3aa16e25a4d2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.919881 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.919645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hxfp\" (UniqueName: \"kubernetes.io/projected/112c52fa-a621-4eca-adda-3aa16e25a4d2-kube-api-access-5hxfp\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.919997 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.919970 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/112c52fa-a621-4eca-adda-3aa16e25a4d2-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.920181 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.919986 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/112c52fa-a621-4eca-adda-3aa16e25a4d2-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.920296 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.920205 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/112c52fa-a621-4eca-adda-3aa16e25a4d2-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.922055 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.922036 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/112c52fa-a621-4eca-adda-3aa16e25a4d2-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.922500 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.922484 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112c52fa-a621-4eca-adda-3aa16e25a4d2-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:18.929186 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:18.929164 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hxfp\" (UniqueName: \"kubernetes.io/projected/112c52fa-a621-4eca-adda-3aa16e25a4d2-kube-api-access-5hxfp\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-7f626\" (UID: \"112c52fa-a621-4eca-adda-3aa16e25a4d2\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:19.052297 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:19.052212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:19.326637 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:19.326610 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626"] Sep 08 14:27:19.328154 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:27:19.328131 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112c52fa_a621_4eca_adda_3aa16e25a4d2.slice/crio-556282172f4f0a00ee9c42050f9d05cd76bc4c2988dced419b0836faae43dc96 WatchSource:0}: Error finding container 556282172f4f0a00ee9c42050f9d05cd76bc4c2988dced419b0836faae43dc96: Status 404 returned error can't find the container with id 556282172f4f0a00ee9c42050f9d05cd76bc4c2988dced419b0836faae43dc96 Sep 08 14:27:19.329446 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:19.329425 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 14:27:19.722060 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:19.722023 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" event={"ID":"112c52fa-a621-4eca-adda-3aa16e25a4d2","Type":"ContainerStarted","Data":"556282172f4f0a00ee9c42050f9d05cd76bc4c2988dced419b0836faae43dc96"} Sep 08 14:27:20.729111 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:20.729078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" event={"ID":"112c52fa-a621-4eca-adda-3aa16e25a4d2","Type":"ContainerStarted","Data":"97fafbeffefd0732eabaac02152b0644396108452cd9fb414a2243b7ca92c449"} Sep 08 14:27:20.746466 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:20.746413 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" podStartSLOduration=1.6888075489999999 podStartE2EDuration="2.746397925s" totalImagesPullingTime="1.057590376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:27:20.744990095 +0000 UTC m=+653.029307212" watchObservedRunningTime="2026-09-08 14:27:20.746397925 +0000 UTC m=+653.030715084" Sep 08 14:27:29.053079 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:29.053040 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:29.053646 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:29.053105 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:29.065377 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:29.065352 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:27:29.799239 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:27:29.799203 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-7f626" Sep 08 14:29:37.803568 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:37.803533 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68d48c8894-jrdrc"] Sep 08 14:29:37.806332 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:37.806315 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68d48c8894-jrdrc" Sep 08 14:29:37.816529 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:37.816500 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68d48c8894-jrdrc"] Sep 08 14:29:37.881972 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:37.881933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab33ab3a-7f69-4a89-9e69-6923da90e606-tls-cert\") pod \"authorino-68d48c8894-jrdrc\" (UID: \"ab33ab3a-7f69-4a89-9e69-6923da90e606\") " pod="kuadrant-system/authorino-68d48c8894-jrdrc" Sep 08 14:29:37.882140 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:37.881983 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg2fd\" (UniqueName: \"kubernetes.io/projected/ab33ab3a-7f69-4a89-9e69-6923da90e606-kube-api-access-mg2fd\") pod \"authorino-68d48c8894-jrdrc\" (UID: \"ab33ab3a-7f69-4a89-9e69-6923da90e606\") " pod="kuadrant-system/authorino-68d48c8894-jrdrc" Sep 08 14:29:37.983263 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:37.983226 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab33ab3a-7f69-4a89-9e69-6923da90e606-tls-cert\") pod \"authorino-68d48c8894-jrdrc\" (UID: \"ab33ab3a-7f69-4a89-9e69-6923da90e606\") " pod="kuadrant-system/authorino-68d48c8894-jrdrc" Sep 08 14:29:37.983263 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:37.983268 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg2fd\" (UniqueName: \"kubernetes.io/projected/ab33ab3a-7f69-4a89-9e69-6923da90e606-kube-api-access-mg2fd\") pod \"authorino-68d48c8894-jrdrc\" (UID: \"ab33ab3a-7f69-4a89-9e69-6923da90e606\") " pod="kuadrant-system/authorino-68d48c8894-jrdrc" Sep 08 14:29:37.985931 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:37.985898 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ab33ab3a-7f69-4a89-9e69-6923da90e606-tls-cert\") pod \"authorino-68d48c8894-jrdrc\" (UID: \"ab33ab3a-7f69-4a89-9e69-6923da90e606\") " pod="kuadrant-system/authorino-68d48c8894-jrdrc" Sep 08 14:29:37.992270 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:37.992249 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg2fd\" (UniqueName: \"kubernetes.io/projected/ab33ab3a-7f69-4a89-9e69-6923da90e606-kube-api-access-mg2fd\") pod \"authorino-68d48c8894-jrdrc\" (UID: \"ab33ab3a-7f69-4a89-9e69-6923da90e606\") " pod="kuadrant-system/authorino-68d48c8894-jrdrc" Sep 08 14:29:38.116063 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:38.115968 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68d48c8894-jrdrc" Sep 08 14:29:38.388940 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:38.388918 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68d48c8894-jrdrc"] Sep 08 14:29:38.390506 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:29:38.390482 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab33ab3a_7f69_4a89_9e69_6923da90e606.slice/crio-a82fa836e6da9c805160e20ce1934e1c8a0b55624393ff2762d42f980538c24d WatchSource:0}: Error finding container a82fa836e6da9c805160e20ce1934e1c8a0b55624393ff2762d42f980538c24d: Status 404 returned error can't find the container with id a82fa836e6da9c805160e20ce1934e1c8a0b55624393ff2762d42f980538c24d Sep 08 14:29:38.503459 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:38.503421 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68d48c8894-jrdrc" event={"ID":"ab33ab3a-7f69-4a89-9e69-6923da90e606","Type":"ContainerStarted","Data":"a82fa836e6da9c805160e20ce1934e1c8a0b55624393ff2762d42f980538c24d"} Sep 08 14:29:41.523610 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:41.523565 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68d48c8894-jrdrc" event={"ID":"ab33ab3a-7f69-4a89-9e69-6923da90e606","Type":"ContainerStarted","Data":"d7c47d38e5291fa3f1e8a3457cca8aa1bd9200a0f15d5728bc128886bce6ab9b"} Sep 08 14:29:41.540340 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:41.540276 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-68d48c8894-jrdrc" podStartSLOduration=2.04678358 podStartE2EDuration="4.540253702s" totalImagesPullingTime="2.493470122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:29:41.538444056 +0000 UTC m=+793.822761172" watchObservedRunningTime="2026-09-08 14:29:41.540253702 +0000 UTC m=+793.824570818" Sep 08 14:29:41.572450 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:41.572419 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-666cfb8df9-5fngs"] Sep 08 14:29:41.572803 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:41.572721 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-666cfb8df9-5fngs" podUID="03d8c75d-c7e0-4915-9209-6cc7af4bf172" containerName="authorino" containerID="cri-o://a47178f7aacc03117f0ed40014e8c7b735ef259fe5f648bbf874b2dec152957a" gracePeriod=30 Sep 08 14:29:41.829055 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:41.829031 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-666cfb8df9-5fngs" Sep 08 14:29:41.917785 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:41.917751 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rwkb\" (UniqueName: \"kubernetes.io/projected/03d8c75d-c7e0-4915-9209-6cc7af4bf172-kube-api-access-8rwkb\") pod \"03d8c75d-c7e0-4915-9209-6cc7af4bf172\" (UID: \"03d8c75d-c7e0-4915-9209-6cc7af4bf172\") " Sep 08 14:29:41.917957 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:41.917860 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03d8c75d-c7e0-4915-9209-6cc7af4bf172-tls-cert\") pod \"03d8c75d-c7e0-4915-9209-6cc7af4bf172\" (UID: \"03d8c75d-c7e0-4915-9209-6cc7af4bf172\") " Sep 08 14:29:41.920036 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:41.920004 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d8c75d-c7e0-4915-9209-6cc7af4bf172-kube-api-access-8rwkb" (OuterVolumeSpecName: "kube-api-access-8rwkb") pod "03d8c75d-c7e0-4915-9209-6cc7af4bf172" (UID: "03d8c75d-c7e0-4915-9209-6cc7af4bf172"). InnerVolumeSpecName "kube-api-access-8rwkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:29:41.930683 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:41.930658 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03d8c75d-c7e0-4915-9209-6cc7af4bf172-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "03d8c75d-c7e0-4915-9209-6cc7af4bf172" (UID: "03d8c75d-c7e0-4915-9209-6cc7af4bf172"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:29:42.019364 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:42.019324 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rwkb\" (UniqueName: \"kubernetes.io/projected/03d8c75d-c7e0-4915-9209-6cc7af4bf172-kube-api-access-8rwkb\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:29:42.019364 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:42.019363 2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/03d8c75d-c7e0-4915-9209-6cc7af4bf172-tls-cert\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:29:42.531969 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:42.531880 2578 generic.go:356] "Generic (PLEG): container finished" podID="03d8c75d-c7e0-4915-9209-6cc7af4bf172" containerID="a47178f7aacc03117f0ed40014e8c7b735ef259fe5f648bbf874b2dec152957a" exitCode=0 Sep 08 14:29:42.531969 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:42.531934 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-666cfb8df9-5fngs" Sep 08 14:29:42.531969 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:42.531962 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-666cfb8df9-5fngs" event={"ID":"03d8c75d-c7e0-4915-9209-6cc7af4bf172","Type":"ContainerDied","Data":"a47178f7aacc03117f0ed40014e8c7b735ef259fe5f648bbf874b2dec152957a"} Sep 08 14:29:42.532558 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:42.532002 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-666cfb8df9-5fngs" event={"ID":"03d8c75d-c7e0-4915-9209-6cc7af4bf172","Type":"ContainerDied","Data":"6f2bfe10436d8186e08bf07c8b58a7b839c972597699b999faad0710ad7c7946"} Sep 08 14:29:42.532558 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:42.532024 2578 scope.go:117] "RemoveContainer" containerID="a47178f7aacc03117f0ed40014e8c7b735ef259fe5f648bbf874b2dec152957a" Sep 08 14:29:42.541665 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:42.541643 2578 scope.go:117] "RemoveContainer" containerID="a47178f7aacc03117f0ed40014e8c7b735ef259fe5f648bbf874b2dec152957a" Sep 08 14:29:42.542110 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:29:42.541988 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a47178f7aacc03117f0ed40014e8c7b735ef259fe5f648bbf874b2dec152957a\": container with ID starting with a47178f7aacc03117f0ed40014e8c7b735ef259fe5f648bbf874b2dec152957a not found: ID does not exist" containerID="a47178f7aacc03117f0ed40014e8c7b735ef259fe5f648bbf874b2dec152957a" Sep 08 14:29:42.542110 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:42.542023 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a47178f7aacc03117f0ed40014e8c7b735ef259fe5f648bbf874b2dec152957a"} err="failed to get container status \"a47178f7aacc03117f0ed40014e8c7b735ef259fe5f648bbf874b2dec152957a\": rpc error: code = NotFound desc = could not find container \"a47178f7aacc03117f0ed40014e8c7b735ef259fe5f648bbf874b2dec152957a\": container with ID starting with a47178f7aacc03117f0ed40014e8c7b735ef259fe5f648bbf874b2dec152957a not found: ID does not exist" Sep 08 14:29:42.552989 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:42.552959 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-666cfb8df9-5fngs"] Sep 08 14:29:42.563123 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:42.563093 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-666cfb8df9-5fngs"] Sep 08 14:29:44.243136 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:29:44.243103 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03d8c75d-c7e0-4915-9209-6cc7af4bf172" path="/var/lib/kubelet/pods/03d8c75d-c7e0-4915-9209-6cc7af4bf172/volumes" Sep 08 14:30:00.868649 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:00.868556 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7907-8428a9"] Sep 08 14:30:00.869164 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:00.869143 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d8c75d-c7e0-4915-9209-6cc7af4bf172" containerName="authorino" Sep 08 14:30:00.869242 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:00.869169 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d8c75d-c7e0-4915-9209-6cc7af4bf172" containerName="authorino" Sep 08 14:30:00.869297 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:00.869275 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="03d8c75d-c7e0-4915-9209-6cc7af4bf172" containerName="authorino" Sep 08 14:30:00.872613 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:00.872593 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7907-8428a9" Sep 08 14:30:00.875187 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:00.875162 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qcp6n\"" Sep 08 14:30:00.878476 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:00.878440 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7907-8428a9"] Sep 08 14:30:00.994047 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:00.994010 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5kdt\" (UniqueName: \"kubernetes.io/projected/49fd2e7c-ae9f-44a8-9970-b666c05a1ad3-kube-api-access-q5kdt\") pod \"test-curl-7907-8428a9\" (UID: \"49fd2e7c-ae9f-44a8-9970-b666c05a1ad3\") " pod="opendatahub/test-curl-7907-8428a9" Sep 08 14:30:01.094636 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:01.094599 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5kdt\" (UniqueName: \"kubernetes.io/projected/49fd2e7c-ae9f-44a8-9970-b666c05a1ad3-kube-api-access-q5kdt\") pod \"test-curl-7907-8428a9\" (UID: \"49fd2e7c-ae9f-44a8-9970-b666c05a1ad3\") " pod="opendatahub/test-curl-7907-8428a9" Sep 08 14:30:01.102771 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:01.102717 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5kdt\" (UniqueName: \"kubernetes.io/projected/49fd2e7c-ae9f-44a8-9970-b666c05a1ad3-kube-api-access-q5kdt\") pod \"test-curl-7907-8428a9\" (UID: \"49fd2e7c-ae9f-44a8-9970-b666c05a1ad3\") " pod="opendatahub/test-curl-7907-8428a9" Sep 08 14:30:01.183636 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:01.183546 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7907-8428a9" Sep 08 14:30:01.453749 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:01.453696 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7907-8428a9"] Sep 08 14:30:01.455502 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:30:01.455479 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49fd2e7c_ae9f_44a8_9970_b666c05a1ad3.slice/crio-66e81257d9060c29a4ecefab9e34a6109d383505f413d7c9d560627d505bb679 WatchSource:0}: Error finding container 66e81257d9060c29a4ecefab9e34a6109d383505f413d7c9d560627d505bb679: Status 404 returned error can't find the container with id 66e81257d9060c29a4ecefab9e34a6109d383505f413d7c9d560627d505bb679 Sep 08 14:30:01.638562 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:01.638527 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7907-8428a9" event={"ID":"49fd2e7c-ae9f-44a8-9970-b666c05a1ad3","Type":"ContainerStarted","Data":"66e81257d9060c29a4ecefab9e34a6109d383505f413d7c9d560627d505bb679"} Sep 08 14:30:03.499455 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.499420 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm"] Sep 08 14:30:03.504323 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.504301 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:03.506766 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.506715 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-75b8cd-serving-cert\"" Sep 08 14:30:03.515885 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.515849 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm"] Sep 08 14:30:03.617353 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.617316 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n68mx\" (UniqueName: \"kubernetes.io/projected/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-kube-api-access-n68mx\") pod \"maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:03.617515 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.617400 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-maas-api-tls\") pod \"maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:03.617515 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.617419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:03.617515 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.617433 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-metrics-certs\") pod \"maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:03.718336 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.718278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n68mx\" (UniqueName: \"kubernetes.io/projected/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-kube-api-access-n68mx\") pod \"maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:03.718551 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.718412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-maas-api-tls\") pod \"maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:03.718551 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.718436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:03.718551 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.718470 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-metrics-certs\") pod \"maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:03.718773 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:30:03.718554 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-serving-cert: secret "maas-api-e2e-inf-a-75b8cd-serving-cert" not found Sep 08 14:30:03.718773 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:30:03.718617 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-maas-api-tls podName:07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d nodeName:}" failed. No retries permitted until 2026-09-08 14:30:04.218601657 +0000 UTC m=+816.502918751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-maas-api-tls") pod "maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" (UID: "07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d") : secret "maas-api-e2e-inf-a-75b8cd-serving-cert" not found Sep 08 14:30:03.719146 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.719124 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:03.721186 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.721164 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-metrics-certs\") pod \"maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:03.727024 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:03.727004 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n68mx\" (UniqueName: \"kubernetes.io/projected/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-kube-api-access-n68mx\") pod \"maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:04.222799 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.222761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-maas-api-tls\") pod \"maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:04.225457 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.225427 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-maas-api-tls\") pod \"maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:04.288766 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.288687 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2"] Sep 08 14:30:04.298850 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.298817 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h"] Sep 08 14:30:04.300218 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.300194 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" Sep 08 14:30:04.303148 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.303127 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-75b8cd\"" Sep 08 14:30:04.303347 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.303323 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-75b8cd-dockercfg-c2bj5\"" Sep 08 14:30:04.304528 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.304503 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2"] Sep 08 14:30:04.304635 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.304550 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h"] Sep 08 14:30:04.304635 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.304610 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" Sep 08 14:30:04.418807 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.418773 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:04.425237 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.425207 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9v4x\" (UniqueName: \"kubernetes.io/projected/bf5f5318-9253-407d-adfc-ea741d4e26ca-kube-api-access-x9v4x\") pod \"payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h\" (UID: \"bf5f5318-9253-407d-adfc-ea741d4e26ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" Sep 08 14:30:04.425388 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.425327 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kldfv\" (UniqueName: \"kubernetes.io/projected/9ee38e92-c346-4bd7-9555-b8ec8253f10a-kube-api-access-kldfv\") pod \"payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2\" (UID: \"9ee38e92-c346-4bd7-9555-b8ec8253f10a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" Sep 08 14:30:04.425459 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.425411 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5f5318-9253-407d-adfc-ea741d4e26ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h\" (UID: \"bf5f5318-9253-407d-adfc-ea741d4e26ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" Sep 08 14:30:04.425459 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.425450 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee38e92-c346-4bd7-9555-b8ec8253f10a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2\" (UID: \"9ee38e92-c346-4bd7-9555-b8ec8253f10a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" Sep 08 14:30:04.527930 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.526546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kldfv\" (UniqueName: \"kubernetes.io/projected/9ee38e92-c346-4bd7-9555-b8ec8253f10a-kube-api-access-kldfv\") pod \"payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2\" (UID: \"9ee38e92-c346-4bd7-9555-b8ec8253f10a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" Sep 08 14:30:04.527930 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.526648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5f5318-9253-407d-adfc-ea741d4e26ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h\" (UID: \"bf5f5318-9253-407d-adfc-ea741d4e26ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" Sep 08 14:30:04.527930 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.526683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee38e92-c346-4bd7-9555-b8ec8253f10a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2\" (UID: \"9ee38e92-c346-4bd7-9555-b8ec8253f10a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" Sep 08 14:30:04.527930 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.526765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9v4x\" (UniqueName: \"kubernetes.io/projected/bf5f5318-9253-407d-adfc-ea741d4e26ca-kube-api-access-x9v4x\") pod \"payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h\" (UID: \"bf5f5318-9253-407d-adfc-ea741d4e26ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" Sep 08 14:30:04.527930 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.527807 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5f5318-9253-407d-adfc-ea741d4e26ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h\" (UID: \"bf5f5318-9253-407d-adfc-ea741d4e26ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" Sep 08 14:30:04.527930 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.527807 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee38e92-c346-4bd7-9555-b8ec8253f10a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2\" (UID: \"9ee38e92-c346-4bd7-9555-b8ec8253f10a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" Sep 08 14:30:04.537650 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.537617 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9v4x\" (UniqueName: \"kubernetes.io/projected/bf5f5318-9253-407d-adfc-ea741d4e26ca-kube-api-access-x9v4x\") pod \"payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h\" (UID: \"bf5f5318-9253-407d-adfc-ea741d4e26ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" Sep 08 14:30:04.538212 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.538187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kldfv\" (UniqueName: \"kubernetes.io/projected/9ee38e92-c346-4bd7-9555-b8ec8253f10a-kube-api-access-kldfv\") pod \"payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2\" (UID: \"9ee38e92-c346-4bd7-9555-b8ec8253f10a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" Sep 08 14:30:04.617971 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.617932 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" Sep 08 14:30:04.626587 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.626549 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" Sep 08 14:30:04.715187 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:30:04.715155 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d0a4c1_29ea_4a8d_8c2a_6470d37fe19d.slice/crio-1d44d36c38675487565694dd02f7a83da6b72d17476f9b502d00ff3763b369c7 WatchSource:0}: Error finding container 1d44d36c38675487565694dd02f7a83da6b72d17476f9b502d00ff3763b369c7: Status 404 returned error can't find the container with id 1d44d36c38675487565694dd02f7a83da6b72d17476f9b502d00ff3763b369c7 Sep 08 14:30:04.717187 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.717146 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm"] Sep 08 14:30:04.897458 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.897421 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2"] Sep 08 14:30:04.899811 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:30:04.899781 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee38e92_c346_4bd7_9555_b8ec8253f10a.slice/crio-86026de0da41a2c296f7d64d34ae7bf8c09570606d16fa7ad2e6266d02de8617 WatchSource:0}: Error finding container 86026de0da41a2c296f7d64d34ae7bf8c09570606d16fa7ad2e6266d02de8617: Status 404 returned error can't find the container with id 86026de0da41a2c296f7d64d34ae7bf8c09570606d16fa7ad2e6266d02de8617 Sep 08 14:30:04.919928 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:04.919830 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h"] Sep 08 14:30:04.921180 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:30:04.921155 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf5f5318_9253_407d_adfc_ea741d4e26ca.slice/crio-e2da609769ca96b5dec91b4a9d769f08f2b91fa3cf69d1900c528537ce0eb1ba WatchSource:0}: Error finding container e2da609769ca96b5dec91b4a9d769f08f2b91fa3cf69d1900c528537ce0eb1ba: Status 404 returned error can't find the container with id e2da609769ca96b5dec91b4a9d769f08f2b91fa3cf69d1900c528537ce0eb1ba Sep 08 14:30:05.666137 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:05.666097 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" event={"ID":"bf5f5318-9253-407d-adfc-ea741d4e26ca","Type":"ContainerStarted","Data":"fbb5f8bb4729c203b0233d9635a7f7e80bda45e58b95b62e3f096ae2f3bce910"} Sep 08 14:30:05.666589 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:05.666154 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" event={"ID":"bf5f5318-9253-407d-adfc-ea741d4e26ca","Type":"ContainerStarted","Data":"e2da609769ca96b5dec91b4a9d769f08f2b91fa3cf69d1900c528537ce0eb1ba"} Sep 08 14:30:05.666589 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:05.666219 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" Sep 08 14:30:05.668011 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:05.667944 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" event={"ID":"9ee38e92-c346-4bd7-9555-b8ec8253f10a","Type":"ContainerStarted","Data":"091fe0d1a62a946f4f189cad9ba769b3dc9de0b4ff702fcd98aeac72a6bd3b6e"} Sep 08 14:30:05.668011 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:05.667981 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" event={"ID":"9ee38e92-c346-4bd7-9555-b8ec8253f10a","Type":"ContainerStarted","Data":"86026de0da41a2c296f7d64d34ae7bf8c09570606d16fa7ad2e6266d02de8617"} Sep 08 14:30:05.668325 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:05.668022 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" Sep 08 14:30:05.669649 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:05.669630 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" event={"ID":"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d","Type":"ContainerStarted","Data":"96e236a9ea01bf9291ccd02573f55e8d31bf35f0c0709e5b3bbde2799361b7bf"} Sep 08 14:30:05.669800 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:05.669661 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" event={"ID":"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d","Type":"ContainerStarted","Data":"1d44d36c38675487565694dd02f7a83da6b72d17476f9b502d00ff3763b369c7"} Sep 08 14:30:05.669800 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:05.669765 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:05.683720 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:05.683681 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" podStartSLOduration=1.683669236 podStartE2EDuration="1.683669236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:30:05.681372594 +0000 UTC m=+817.965689713" watchObservedRunningTime="2026-09-08 14:30:05.683669236 +0000 UTC m=+817.967986383" Sep 08 14:30:05.698012 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:05.697968 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" podStartSLOduration=1.697957353 podStartE2EDuration="1.697957353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:30:05.695550942 +0000 UTC m=+817.979868078" watchObservedRunningTime="2026-09-08 14:30:05.697957353 +0000 UTC m=+817.982274462" Sep 08 14:30:05.712833 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:05.712774 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" podStartSLOduration=1.979619437 podStartE2EDuration="2.712723586s" totalImagesPullingTime="733.104149ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:30:05.711301846 +0000 UTC m=+817.995618962" watchObservedRunningTime="2026-09-08 14:30:05.712723586 +0000 UTC m=+817.997040698" Sep 08 14:30:11.041405 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.041373 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq"] Sep 08 14:30:11.046309 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.046288 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.048719 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.048694 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-32df8e95-serving-cert\"" Sep 08 14:30:11.055337 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.055315 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq"] Sep 08 14:30:11.194715 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.194674 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b85939c-b570-4e78-989f-b3f5f11b5d3d-openshift-service-ca\") pod \"maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.194926 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.194818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzmbl\" (UniqueName: \"kubernetes.io/projected/4b85939c-b570-4e78-989f-b3f5f11b5d3d-kube-api-access-fzmbl\") pod \"maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.194926 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.194867 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b85939c-b570-4e78-989f-b3f5f11b5d3d-maas-api-tls\") pod \"maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.194926 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.194921 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b85939c-b570-4e78-989f-b3f5f11b5d3d-metrics-certs\") pod \"maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.296294 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.296217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzmbl\" (UniqueName: \"kubernetes.io/projected/4b85939c-b570-4e78-989f-b3f5f11b5d3d-kube-api-access-fzmbl\") pod \"maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.296294 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.296282 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b85939c-b570-4e78-989f-b3f5f11b5d3d-maas-api-tls\") pod \"maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.296527 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.296324 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b85939c-b570-4e78-989f-b3f5f11b5d3d-metrics-certs\") pod \"maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.296527 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.296377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b85939c-b570-4e78-989f-b3f5f11b5d3d-openshift-service-ca\") pod \"maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.297113 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.297087 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b85939c-b570-4e78-989f-b3f5f11b5d3d-openshift-service-ca\") pod \"maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.299101 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.299056 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b85939c-b570-4e78-989f-b3f5f11b5d3d-maas-api-tls\") pod \"maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.299217 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.299118 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b85939c-b570-4e78-989f-b3f5f11b5d3d-metrics-certs\") pod \"maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.304558 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.304528 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzmbl\" (UniqueName: \"kubernetes.io/projected/4b85939c-b570-4e78-989f-b3f5f11b5d3d-kube-api-access-fzmbl\") pod \"maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.362494 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.362452 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:11.685099 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:11.685023 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:30:12.257947 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:12.257924 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq"] Sep 08 14:30:12.259207 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:30:12.259178 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b85939c_b570_4e78_989f_b3f5f11b5d3d.slice/crio-6fcac004725fa07db728be55db437af1871ef47eed1f1726abfdcf73fffd77ee WatchSource:0}: Error finding container 6fcac004725fa07db728be55db437af1871ef47eed1f1726abfdcf73fffd77ee: Status 404 returned error can't find the container with id 6fcac004725fa07db728be55db437af1871ef47eed1f1726abfdcf73fffd77ee Sep 08 14:30:12.719023 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:12.718990 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7907-8428a9" event={"ID":"49fd2e7c-ae9f-44a8-9970-b666c05a1ad3","Type":"ContainerStarted","Data":"f486adca43541ef520acefc7105f8fb48fb7c0635891ca0b808625ae392d9da6"} Sep 08 14:30:12.720790 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:12.720765 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" event={"ID":"4b85939c-b570-4e78-989f-b3f5f11b5d3d","Type":"ContainerStarted","Data":"98b894fe3bfbe79e8dc1b650568a0571277175e6e9e0da80c2eb18f08c12929b"} Sep 08 14:30:12.720904 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:12.720819 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:12.720904 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:12.720837 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" event={"ID":"4b85939c-b570-4e78-989f-b3f5f11b5d3d","Type":"ContainerStarted","Data":"6fcac004725fa07db728be55db437af1871ef47eed1f1726abfdcf73fffd77ee"} Sep 08 14:30:12.732261 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:12.731983 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7907-8428a9" podStartSLOduration=2.116771116 podStartE2EDuration="12.731965342s" totalImagesPullingTime="10.615194226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:30:12.731412461 +0000 UTC m=+825.015729575" watchObservedRunningTime="2026-09-08 14:30:12.731965342 +0000 UTC m=+825.016282459" Sep 08 14:30:12.749592 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:12.749544 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" podStartSLOduration=1.358558804 podStartE2EDuration="1.749528779s" totalImagesPullingTime="390.969975ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:30:12.746669769 +0000 UTC m=+825.030986897" watchObservedRunningTime="2026-09-08 14:30:12.749528779 +0000 UTC m=+825.033845893" Sep 08 14:30:13.343051 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:13.343016 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7907-8428a9"] Sep 08 14:30:13.349664 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:13.349638 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7907-8428a9"] Sep 08 14:30:13.727123 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:30:13.727093 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7907-8428a9" container="test-curl-7907-8428a9" Sep 08 14:30:14.733034 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:14.732992 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7907-8428a9" podUID="49fd2e7c-ae9f-44a8-9970-b666c05a1ad3" containerName="test-curl-7907-8428a9" containerID="cri-o://f486adca43541ef520acefc7105f8fb48fb7c0635891ca0b808625ae392d9da6" gracePeriod=2 Sep 08 14:30:16.680842 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:16.680812 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" Sep 08 14:30:16.681246 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:16.680886 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" Sep 08 14:30:16.930027 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:16.930007 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7907-8428a9" Sep 08 14:30:16.942779 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:16.942717 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5kdt\" (UniqueName: \"kubernetes.io/projected/49fd2e7c-ae9f-44a8-9970-b666c05a1ad3-kube-api-access-q5kdt\") pod \"49fd2e7c-ae9f-44a8-9970-b666c05a1ad3\" (UID: \"49fd2e7c-ae9f-44a8-9970-b666c05a1ad3\") " Sep 08 14:30:16.944683 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:16.944661 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49fd2e7c-ae9f-44a8-9970-b666c05a1ad3-kube-api-access-q5kdt" (OuterVolumeSpecName: "kube-api-access-q5kdt") pod "49fd2e7c-ae9f-44a8-9970-b666c05a1ad3" (UID: "49fd2e7c-ae9f-44a8-9970-b666c05a1ad3"). InnerVolumeSpecName "kube-api-access-q5kdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:30:17.043800 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:17.043761 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5kdt\" (UniqueName: \"kubernetes.io/projected/49fd2e7c-ae9f-44a8-9970-b666c05a1ad3-kube-api-access-q5kdt\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:30:17.753645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:17.753610 2578 generic.go:356] "Generic (PLEG): container finished" podID="49fd2e7c-ae9f-44a8-9970-b666c05a1ad3" containerID="f486adca43541ef520acefc7105f8fb48fb7c0635891ca0b808625ae392d9da6" exitCode=137 Sep 08 14:30:17.754153 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:17.753652 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7907-8428a9" Sep 08 14:30:17.754153 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:17.753712 2578 scope.go:117] "RemoveContainer" containerID="f486adca43541ef520acefc7105f8fb48fb7c0635891ca0b808625ae392d9da6" Sep 08 14:30:17.763497 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:17.763482 2578 scope.go:117] "RemoveContainer" containerID="f486adca43541ef520acefc7105f8fb48fb7c0635891ca0b808625ae392d9da6" Sep 08 14:30:17.763771 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:30:17.763738 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7907-8428a9\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="49fd2e7c-ae9f-44a8-9970-b666c05a1ad3" pod="opendatahub/test-curl-7907-8428a9" Sep 08 14:30:17.763867 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:30:17.763824 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f486adca43541ef520acefc7105f8fb48fb7c0635891ca0b808625ae392d9da6\": container with ID starting with f486adca43541ef520acefc7105f8fb48fb7c0635891ca0b808625ae392d9da6 not found: ID does not exist" containerID="f486adca43541ef520acefc7105f8fb48fb7c0635891ca0b808625ae392d9da6" Sep 08 14:30:17.763867 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:17.763846 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f486adca43541ef520acefc7105f8fb48fb7c0635891ca0b808625ae392d9da6"} err="failed to get container status \"f486adca43541ef520acefc7105f8fb48fb7c0635891ca0b808625ae392d9da6\": rpc error: code = NotFound desc = could not find container \"f486adca43541ef520acefc7105f8fb48fb7c0635891ca0b808625ae392d9da6\": container with ID starting with f486adca43541ef520acefc7105f8fb48fb7c0635891ca0b808625ae392d9da6 not found: ID does not exist" Sep 08 14:30:18.241064 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:30:18.241021 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7907-8428a9\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="49fd2e7c-ae9f-44a8-9970-b666c05a1ad3" pod="opendatahub/test-curl-7907-8428a9" Sep 08 14:30:18.244449 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:18.244417 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49fd2e7c-ae9f-44a8-9970-b666c05a1ad3" path="/var/lib/kubelet/pods/49fd2e7c-ae9f-44a8-9970-b666c05a1ad3/volumes" Sep 08 14:30:18.733209 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:18.733177 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:23.671159 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.671118 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl"] Sep 08 14:30:23.671820 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.671799 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49fd2e7c-ae9f-44a8-9970-b666c05a1ad3" containerName="test-curl-7907-8428a9" Sep 08 14:30:23.671873 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.671823 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="49fd2e7c-ae9f-44a8-9970-b666c05a1ad3" containerName="test-curl-7907-8428a9" Sep 08 14:30:23.671977 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.671966 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="49fd2e7c-ae9f-44a8-9970-b666c05a1ad3" containerName="test-curl-7907-8428a9" Sep 08 14:30:23.675552 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.675532 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:23.677917 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.677893 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-47d144-serving-cert\"" Sep 08 14:30:23.686068 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.686046 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl"] Sep 08 14:30:23.705580 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.705546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-maas-api-tls\") pod \"maas-api-e2e-autores-47d144-546f6bcc4b-rngcl\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:23.705700 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.705625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03b0eee-8606-4423-9a4c-5e944be993b2-openshift-service-ca\") pod \"maas-api-e2e-autores-47d144-546f6bcc4b-rngcl\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:23.705800 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.705779 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6mqn\" (UniqueName: \"kubernetes.io/projected/d03b0eee-8606-4423-9a4c-5e944be993b2-kube-api-access-m6mqn\") pod \"maas-api-e2e-autores-47d144-546f6bcc4b-rngcl\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:23.705842 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.705828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-metrics-certs\") pod \"maas-api-e2e-autores-47d144-546f6bcc4b-rngcl\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:23.807044 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.807006 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-maas-api-tls\") pod \"maas-api-e2e-autores-47d144-546f6bcc4b-rngcl\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:23.807258 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.807061 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03b0eee-8606-4423-9a4c-5e944be993b2-openshift-service-ca\") pod \"maas-api-e2e-autores-47d144-546f6bcc4b-rngcl\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:23.807258 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.807147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6mqn\" (UniqueName: \"kubernetes.io/projected/d03b0eee-8606-4423-9a4c-5e944be993b2-kube-api-access-m6mqn\") pod \"maas-api-e2e-autores-47d144-546f6bcc4b-rngcl\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:23.807258 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:30:23.807160 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-47d144-serving-cert: secret "maas-api-e2e-autores-47d144-serving-cert" not found Sep 08 14:30:23.807258 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.807198 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-metrics-certs\") pod \"maas-api-e2e-autores-47d144-546f6bcc4b-rngcl\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:23.807258 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:30:23.807258 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-maas-api-tls podName:d03b0eee-8606-4423-9a4c-5e944be993b2 nodeName:}" failed. No retries permitted until 2026-09-08 14:30:24.307237446 +0000 UTC m=+836.591554539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-maas-api-tls") pod "maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" (UID: "d03b0eee-8606-4423-9a4c-5e944be993b2") : secret "maas-api-e2e-autores-47d144-serving-cert" not found Sep 08 14:30:23.807806 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.807777 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03b0eee-8606-4423-9a4c-5e944be993b2-openshift-service-ca\") pod \"maas-api-e2e-autores-47d144-546f6bcc4b-rngcl\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:23.809718 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.809694 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-metrics-certs\") pod \"maas-api-e2e-autores-47d144-546f6bcc4b-rngcl\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:23.815356 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:23.815335 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6mqn\" (UniqueName: \"kubernetes.io/projected/d03b0eee-8606-4423-9a4c-5e944be993b2-kube-api-access-m6mqn\") pod \"maas-api-e2e-autores-47d144-546f6bcc4b-rngcl\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:24.313180 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.313138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-maas-api-tls\") pod \"maas-api-e2e-autores-47d144-546f6bcc4b-rngcl\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:24.315627 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.315606 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-maas-api-tls\") pod \"maas-api-e2e-autores-47d144-546f6bcc4b-rngcl\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:24.356540 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.356507 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6"] Sep 08 14:30:24.360369 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.360346 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" Sep 08 14:30:24.362577 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.362558 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-47d144\"" Sep 08 14:30:24.362666 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.362559 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-47d144-dockercfg-nxghc\"" Sep 08 14:30:24.371293 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.371242 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6"] Sep 08 14:30:24.413640 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.413605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea56ce4-2712-466f-86d2-736ca9eb3956-plugins-config-volume\") pod \"payload-processing-e2e-autores-47d144-bf58f96d5-gztf6\" (UID: \"bea56ce4-2712-466f-86d2-736ca9eb3956\") " pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" Sep 08 14:30:24.413826 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.413661 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thlfs\" (UniqueName: \"kubernetes.io/projected/bea56ce4-2712-466f-86d2-736ca9eb3956-kube-api-access-thlfs\") pod \"payload-processing-e2e-autores-47d144-bf58f96d5-gztf6\" (UID: \"bea56ce4-2712-466f-86d2-736ca9eb3956\") " pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" Sep 08 14:30:24.515175 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.515145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thlfs\" (UniqueName: \"kubernetes.io/projected/bea56ce4-2712-466f-86d2-736ca9eb3956-kube-api-access-thlfs\") pod \"payload-processing-e2e-autores-47d144-bf58f96d5-gztf6\" (UID: \"bea56ce4-2712-466f-86d2-736ca9eb3956\") " pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" Sep 08 14:30:24.515338 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.515253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea56ce4-2712-466f-86d2-736ca9eb3956-plugins-config-volume\") pod \"payload-processing-e2e-autores-47d144-bf58f96d5-gztf6\" (UID: \"bea56ce4-2712-466f-86d2-736ca9eb3956\") " pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" Sep 08 14:30:24.515896 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.515878 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea56ce4-2712-466f-86d2-736ca9eb3956-plugins-config-volume\") pod \"payload-processing-e2e-autores-47d144-bf58f96d5-gztf6\" (UID: \"bea56ce4-2712-466f-86d2-736ca9eb3956\") " pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" Sep 08 14:30:24.523255 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.523224 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thlfs\" (UniqueName: \"kubernetes.io/projected/bea56ce4-2712-466f-86d2-736ca9eb3956-kube-api-access-thlfs\") pod \"payload-processing-e2e-autores-47d144-bf58f96d5-gztf6\" (UID: \"bea56ce4-2712-466f-86d2-736ca9eb3956\") " pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" Sep 08 14:30:24.589123 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.589035 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:24.673241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.673209 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" Sep 08 14:30:24.866721 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.866694 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl"] Sep 08 14:30:24.867583 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:30:24.867562 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03b0eee_8606_4423_9a4c_5e944be993b2.slice/crio-c95abe2ae04b0d028e502bb8a83ea0ff8a1e5b1da114d41a84b55fa6ddb33f44 WatchSource:0}: Error finding container c95abe2ae04b0d028e502bb8a83ea0ff8a1e5b1da114d41a84b55fa6ddb33f44: Status 404 returned error can't find the container with id c95abe2ae04b0d028e502bb8a83ea0ff8a1e5b1da114d41a84b55fa6ddb33f44 Sep 08 14:30:24.954833 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:24.954795 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6"] Sep 08 14:30:24.955432 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:30:24.955395 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea56ce4_2712_466f_86d2_736ca9eb3956.slice/crio-871650fa16d1ca02d8e672d734a80c08bf250eeed7f97bc2935ac3ba49d15fb0 WatchSource:0}: Error finding container 871650fa16d1ca02d8e672d734a80c08bf250eeed7f97bc2935ac3ba49d15fb0: Status 404 returned error can't find the container with id 871650fa16d1ca02d8e672d734a80c08bf250eeed7f97bc2935ac3ba49d15fb0 Sep 08 14:30:25.811958 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:25.811914 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" event={"ID":"bea56ce4-2712-466f-86d2-736ca9eb3956","Type":"ContainerStarted","Data":"ed2efaa036162e3075f78ae7f9744e533c54e63ccafb7eff99a3da8379e38afc"} Sep 08 14:30:25.812411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:25.811981 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" event={"ID":"bea56ce4-2712-466f-86d2-736ca9eb3956","Type":"ContainerStarted","Data":"871650fa16d1ca02d8e672d734a80c08bf250eeed7f97bc2935ac3ba49d15fb0"} Sep 08 14:30:25.812411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:25.812022 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" Sep 08 14:30:25.813305 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:25.813279 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" event={"ID":"d03b0eee-8606-4423-9a4c-5e944be993b2","Type":"ContainerStarted","Data":"c95abe2ae04b0d028e502bb8a83ea0ff8a1e5b1da114d41a84b55fa6ddb33f44"} Sep 08 14:30:25.830250 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:25.830203 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" podStartSLOduration=1.830189471 podStartE2EDuration="1.830189471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:30:25.82913473 +0000 UTC m=+838.113451846" watchObservedRunningTime="2026-09-08 14:30:25.830189471 +0000 UTC m=+838.114506586" Sep 08 14:30:30.853648 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:30.853614 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" event={"ID":"d03b0eee-8606-4423-9a4c-5e944be993b2","Type":"ContainerStarted","Data":"ecec1c1e975b4ba33cc187e798cc3ae0ada70524294a0528abef24f977ea75b2"} Sep 08 14:30:30.854072 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:30.853753 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:30.870244 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:30.870188 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" podStartSLOduration=2.575881953 podStartE2EDuration="7.870168527s" totalImagesPullingTime="5.294286574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:30:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:30:30.869195021 +0000 UTC m=+843.153512137" watchObservedRunningTime="2026-09-08 14:30:30.870168527 +0000 UTC m=+843.154485643" Sep 08 14:30:32.700318 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.700282 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl"] Sep 08 14:30:32.704937 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.704915 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:32.707223 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.707202 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2a2de3-serving-cert\"" Sep 08 14:30:32.714422 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.714398 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl"] Sep 08 14:30:32.795985 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.795950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-metrics-certs\") pod \"maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:32.796163 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.796000 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9jn9\" (UniqueName: \"kubernetes.io/projected/79a0c411-4969-4d42-811b-44a067de2c30-kube-api-access-b9jn9\") pod \"maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:32.796163 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.796128 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a0c411-4969-4d42-811b-44a067de2c30-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:32.796251 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.796164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:32.897266 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.897231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a0c411-4969-4d42-811b-44a067de2c30-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:32.897433 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.897275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:32.897433 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.897339 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-metrics-certs\") pod \"maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:32.897433 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.897365 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9jn9\" (UniqueName: \"kubernetes.io/projected/79a0c411-4969-4d42-811b-44a067de2c30-kube-api-access-b9jn9\") pod \"maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:32.897542 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:30:32.897455 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-serving-cert: secret "maas-api-e2e-inf-b-2a2de3-serving-cert" not found Sep 08 14:30:32.897542 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:30:32.897534 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-maas-api-tls podName:79a0c411-4969-4d42-811b-44a067de2c30 nodeName:}" failed. No retries permitted until 2026-09-08 14:30:33.397511248 +0000 UTC m=+845.681828362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-maas-api-tls") pod "maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" (UID: "79a0c411-4969-4d42-811b-44a067de2c30") : secret "maas-api-e2e-inf-b-2a2de3-serving-cert" not found Sep 08 14:30:32.897974 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.897949 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a0c411-4969-4d42-811b-44a067de2c30-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:32.899786 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.899771 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-metrics-certs\") pod \"maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:32.905958 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:32.905937 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9jn9\" (UniqueName: \"kubernetes.io/projected/79a0c411-4969-4d42-811b-44a067de2c30-kube-api-access-b9jn9\") pod \"maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:33.401823 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:33.401787 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:33.404414 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:33.404387 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-maas-api-tls\") pod \"maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:33.619184 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:33.619145 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:33.898912 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:33.898878 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl"] Sep 08 14:30:33.899235 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:30:33.899009 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a0c411_4969_4d42_811b_44a067de2c30.slice/crio-76a8c8bdea4d6548b7f01b2d3870414a799b520837e8a6906de42d1005c133ce WatchSource:0}: Error finding container 76a8c8bdea4d6548b7f01b2d3870414a799b520837e8a6906de42d1005c133ce: Status 404 returned error can't find the container with id 76a8c8bdea4d6548b7f01b2d3870414a799b520837e8a6906de42d1005c133ce Sep 08 14:30:34.878420 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:34.878378 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" event={"ID":"79a0c411-4969-4d42-811b-44a067de2c30","Type":"ContainerStarted","Data":"df72b65522604442a2351d536c8f70a6143e8b86e4acadc6d1a11e32d89c7579"} Sep 08 14:30:34.878603 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:34.878444 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" event={"ID":"79a0c411-4969-4d42-811b-44a067de2c30","Type":"ContainerStarted","Data":"76a8c8bdea4d6548b7f01b2d3870414a799b520837e8a6906de42d1005c133ce"} Sep 08 14:30:34.878603 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:34.878522 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:34.895854 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:34.895809 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" podStartSLOduration=2.509684717 podStartE2EDuration="2.895794266s" totalImagesPullingTime="386.109549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:30:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:30:34.894719865 +0000 UTC m=+847.179036980" watchObservedRunningTime="2026-09-08 14:30:34.895794266 +0000 UTC m=+847.180111382" Sep 08 14:30:36.821834 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:36.821798 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" Sep 08 14:30:36.866228 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:36.866196 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:30:40.889616 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:40.889577 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:30:51.773224 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:51.773187 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq"] Sep 08 14:30:51.774071 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:51.774014 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" podUID="4b85939c-b570-4e78-989f-b3f5f11b5d3d" containerName="maas-api" containerID="cri-o://98b894fe3bfbe79e8dc1b650568a0571277175e6e9e0da80c2eb18f08c12929b" gracePeriod=30 Sep 08 14:30:51.981302 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:51.981269 2578 generic.go:356] "Generic (PLEG): container finished" podID="4b85939c-b570-4e78-989f-b3f5f11b5d3d" containerID="98b894fe3bfbe79e8dc1b650568a0571277175e6e9e0da80c2eb18f08c12929b" exitCode=0 Sep 08 14:30:51.981439 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:51.981347 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" event={"ID":"4b85939c-b570-4e78-989f-b3f5f11b5d3d","Type":"ContainerDied","Data":"98b894fe3bfbe79e8dc1b650568a0571277175e6e9e0da80c2eb18f08c12929b"} Sep 08 14:30:52.028149 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.028076 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:52.196467 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.196436 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b85939c-b570-4e78-989f-b3f5f11b5d3d-maas-api-tls\") pod \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " Sep 08 14:30:52.196467 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.196473 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b85939c-b570-4e78-989f-b3f5f11b5d3d-metrics-certs\") pod \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " Sep 08 14:30:52.196671 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.196522 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b85939c-b570-4e78-989f-b3f5f11b5d3d-openshift-service-ca\") pod \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " Sep 08 14:30:52.196671 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.196545 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fzmbl\" (UniqueName: \"kubernetes.io/projected/4b85939c-b570-4e78-989f-b3f5f11b5d3d-kube-api-access-fzmbl\") pod \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\" (UID: \"4b85939c-b570-4e78-989f-b3f5f11b5d3d\") " Sep 08 14:30:52.196973 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.196944 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b85939c-b570-4e78-989f-b3f5f11b5d3d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b85939c-b570-4e78-989f-b3f5f11b5d3d" (UID: "4b85939c-b570-4e78-989f-b3f5f11b5d3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:30:52.199009 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.198986 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b85939c-b570-4e78-989f-b3f5f11b5d3d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b85939c-b570-4e78-989f-b3f5f11b5d3d" (UID: "4b85939c-b570-4e78-989f-b3f5f11b5d3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:30:52.199106 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.199083 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b85939c-b570-4e78-989f-b3f5f11b5d3d-kube-api-access-fzmbl" (OuterVolumeSpecName: "kube-api-access-fzmbl") pod "4b85939c-b570-4e78-989f-b3f5f11b5d3d" (UID: "4b85939c-b570-4e78-989f-b3f5f11b5d3d"). InnerVolumeSpecName "kube-api-access-fzmbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:30:52.199148 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.199101 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b85939c-b570-4e78-989f-b3f5f11b5d3d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4b85939c-b570-4e78-989f-b3f5f11b5d3d" (UID: "4b85939c-b570-4e78-989f-b3f5f11b5d3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:30:52.297877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.297799 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b85939c-b570-4e78-989f-b3f5f11b5d3d-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:30:52.297877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.297827 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b85939c-b570-4e78-989f-b3f5f11b5d3d-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:30:52.297877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.297835 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b85939c-b570-4e78-989f-b3f5f11b5d3d-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:30:52.297877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.297843 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzmbl\" (UniqueName: \"kubernetes.io/projected/4b85939c-b570-4e78-989f-b3f5f11b5d3d-kube-api-access-fzmbl\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:30:52.989104 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.989059 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" event={"ID":"4b85939c-b570-4e78-989f-b3f5f11b5d3d","Type":"ContainerDied","Data":"6fcac004725fa07db728be55db437af1871ef47eed1f1726abfdcf73fffd77ee"} Sep 08 14:30:52.989532 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.989122 2578 scope.go:117] "RemoveContainer" containerID="98b894fe3bfbe79e8dc1b650568a0571277175e6e9e0da80c2eb18f08c12929b" Sep 08 14:30:52.989532 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:52.989072 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq" Sep 08 14:30:53.006795 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:53.006772 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq"] Sep 08 14:30:53.018278 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:53.018252 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-32df8e95-596ccb75b7-fq2sq"] Sep 08 14:30:54.242380 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:54.242336 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b85939c-b570-4e78-989f-b3f5f11b5d3d" path="/var/lib/kubelet/pods/4b85939c-b570-4e78-989f-b3f5f11b5d3d/volumes" Sep 08 14:30:56.541266 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.541231 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9"] Sep 08 14:30:56.541914 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.541897 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b85939c-b570-4e78-989f-b3f5f11b5d3d" containerName="maas-api" Sep 08 14:30:56.541976 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.541919 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b85939c-b570-4e78-989f-b3f5f11b5d3d" containerName="maas-api" Sep 08 14:30:56.542039 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.542029 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b85939c-b570-4e78-989f-b3f5f11b5d3d" containerName="maas-api" Sep 08 14:30:56.545107 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.545085 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:56.547659 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.547638 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0416fe39-serving-cert\"" Sep 08 14:30:56.552994 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.552973 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9"] Sep 08 14:30:56.639514 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.639472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a49177-56d6-4bae-82c7-9007eeb56f15-maas-api-tls\") pod \"maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:56.639514 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.639509 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a49177-56d6-4bae-82c7-9007eeb56f15-openshift-service-ca\") pod \"maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:56.639774 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.639538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a49177-56d6-4bae-82c7-9007eeb56f15-metrics-certs\") pod \"maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:56.639774 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.639602 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4zll\" (UniqueName: \"kubernetes.io/projected/18a49177-56d6-4bae-82c7-9007eeb56f15-kube-api-access-k4zll\") pod \"maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:56.740647 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.740613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a49177-56d6-4bae-82c7-9007eeb56f15-maas-api-tls\") pod \"maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:56.740855 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.740651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a49177-56d6-4bae-82c7-9007eeb56f15-openshift-service-ca\") pod \"maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:56.740855 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.740682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a49177-56d6-4bae-82c7-9007eeb56f15-metrics-certs\") pod \"maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:56.740855 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.740704 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4zll\" (UniqueName: \"kubernetes.io/projected/18a49177-56d6-4bae-82c7-9007eeb56f15-kube-api-access-k4zll\") pod \"maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:56.741337 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.741310 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a49177-56d6-4bae-82c7-9007eeb56f15-openshift-service-ca\") pod \"maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:56.743325 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.743302 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a49177-56d6-4bae-82c7-9007eeb56f15-maas-api-tls\") pod \"maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:56.743402 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.743332 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a49177-56d6-4bae-82c7-9007eeb56f15-metrics-certs\") pod \"maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:56.749169 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.749148 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4zll\" (UniqueName: \"kubernetes.io/projected/18a49177-56d6-4bae-82c7-9007eeb56f15-kube-api-access-k4zll\") pod \"maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:56.858440 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:56.858361 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:30:57.133546 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:57.133518 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9"] Sep 08 14:30:57.134968 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:30:57.134945 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a49177_56d6_4bae_82c7_9007eeb56f15.slice/crio-39bf204cbcf19d9bc97cba47706f356ba109540224755249346d568ba2c4eac4 WatchSource:0}: Error finding container 39bf204cbcf19d9bc97cba47706f356ba109540224755249346d568ba2c4eac4: Status 404 returned error can't find the container with id 39bf204cbcf19d9bc97cba47706f356ba109540224755249346d568ba2c4eac4 Sep 08 14:30:58.025742 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:30:58.025703 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" event={"ID":"18a49177-56d6-4bae-82c7-9007eeb56f15","Type":"ContainerStarted","Data":"39bf204cbcf19d9bc97cba47706f356ba109540224755249346d568ba2c4eac4"} Sep 08 14:31:00.040146 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:00.040108 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" event={"ID":"18a49177-56d6-4bae-82c7-9007eeb56f15","Type":"ContainerStarted","Data":"f32acdb32726dd9505cec24ecb8d124442427632ab8fc003f464f09ed01a8eec"} Sep 08 14:31:00.040606 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:00.040178 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:31:00.057659 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:00.057613 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" podStartSLOduration=1.719593541 podStartE2EDuration="4.05759896s" totalImagesPullingTime="2.338005419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:31:00.055667972 +0000 UTC m=+872.339985098" watchObservedRunningTime="2026-09-08 14:31:00.05759896 +0000 UTC m=+872.341916075" Sep 08 14:31:06.053300 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:06.053271 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:31:27.877839 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:27.877749 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9"] Sep 08 14:31:27.878321 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:27.878035 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" podUID="18a49177-56d6-4bae-82c7-9007eeb56f15" containerName="maas-api" containerID="cri-o://f32acdb32726dd9505cec24ecb8d124442427632ab8fc003f464f09ed01a8eec" gracePeriod=30 Sep 08 14:31:28.151386 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.151361 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:31:28.202280 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.202247 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:31:28.202459 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.202399 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:31:28.211067 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.211045 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:31:28.211067 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.211060 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:31:28.218197 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.218165 2578 generic.go:356] "Generic (PLEG): container finished" podID="18a49177-56d6-4bae-82c7-9007eeb56f15" containerID="f32acdb32726dd9505cec24ecb8d124442427632ab8fc003f464f09ed01a8eec" exitCode=0 Sep 08 14:31:28.218334 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.218218 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" Sep 08 14:31:28.218334 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.218234 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" event={"ID":"18a49177-56d6-4bae-82c7-9007eeb56f15","Type":"ContainerDied","Data":"f32acdb32726dd9505cec24ecb8d124442427632ab8fc003f464f09ed01a8eec"} Sep 08 14:31:28.218334 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.218290 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9" event={"ID":"18a49177-56d6-4bae-82c7-9007eeb56f15","Type":"ContainerDied","Data":"39bf204cbcf19d9bc97cba47706f356ba109540224755249346d568ba2c4eac4"} Sep 08 14:31:28.218334 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.218320 2578 scope.go:117] "RemoveContainer" containerID="f32acdb32726dd9505cec24ecb8d124442427632ab8fc003f464f09ed01a8eec" Sep 08 14:31:28.229460 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.229438 2578 scope.go:117] "RemoveContainer" containerID="f32acdb32726dd9505cec24ecb8d124442427632ab8fc003f464f09ed01a8eec" Sep 08 14:31:28.229840 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:31:28.229810 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f32acdb32726dd9505cec24ecb8d124442427632ab8fc003f464f09ed01a8eec\": container with ID starting with f32acdb32726dd9505cec24ecb8d124442427632ab8fc003f464f09ed01a8eec not found: ID does not exist" containerID="f32acdb32726dd9505cec24ecb8d124442427632ab8fc003f464f09ed01a8eec" Sep 08 14:31:28.229940 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.229848 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f32acdb32726dd9505cec24ecb8d124442427632ab8fc003f464f09ed01a8eec"} err="failed to get container status \"f32acdb32726dd9505cec24ecb8d124442427632ab8fc003f464f09ed01a8eec\": rpc error: code = NotFound desc = could not find container \"f32acdb32726dd9505cec24ecb8d124442427632ab8fc003f464f09ed01a8eec\": container with ID starting with f32acdb32726dd9505cec24ecb8d124442427632ab8fc003f464f09ed01a8eec not found: ID does not exist" Sep 08 14:31:28.249644 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.249613 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a49177-56d6-4bae-82c7-9007eeb56f15-metrics-certs\") pod \"18a49177-56d6-4bae-82c7-9007eeb56f15\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " Sep 08 14:31:28.249862 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.249662 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a49177-56d6-4bae-82c7-9007eeb56f15-maas-api-tls\") pod \"18a49177-56d6-4bae-82c7-9007eeb56f15\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " Sep 08 14:31:28.249862 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.249706 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a49177-56d6-4bae-82c7-9007eeb56f15-openshift-service-ca\") pod \"18a49177-56d6-4bae-82c7-9007eeb56f15\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " Sep 08 14:31:28.250000 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.249955 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4zll\" (UniqueName: \"kubernetes.io/projected/18a49177-56d6-4bae-82c7-9007eeb56f15-kube-api-access-k4zll\") pod \"18a49177-56d6-4bae-82c7-9007eeb56f15\" (UID: \"18a49177-56d6-4bae-82c7-9007eeb56f15\") " Sep 08 14:31:28.250385 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.250281 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a49177-56d6-4bae-82c7-9007eeb56f15-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "18a49177-56d6-4bae-82c7-9007eeb56f15" (UID: "18a49177-56d6-4bae-82c7-9007eeb56f15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:31:28.250385 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.250368 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a49177-56d6-4bae-82c7-9007eeb56f15-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:31:28.252694 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.252619 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a49177-56d6-4bae-82c7-9007eeb56f15-kube-api-access-k4zll" (OuterVolumeSpecName: "kube-api-access-k4zll") pod "18a49177-56d6-4bae-82c7-9007eeb56f15" (UID: "18a49177-56d6-4bae-82c7-9007eeb56f15"). InnerVolumeSpecName "kube-api-access-k4zll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:31:28.252694 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.252687 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a49177-56d6-4bae-82c7-9007eeb56f15-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "18a49177-56d6-4bae-82c7-9007eeb56f15" (UID: "18a49177-56d6-4bae-82c7-9007eeb56f15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:31:28.252968 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.252715 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a49177-56d6-4bae-82c7-9007eeb56f15-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "18a49177-56d6-4bae-82c7-9007eeb56f15" (UID: "18a49177-56d6-4bae-82c7-9007eeb56f15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:31:28.351219 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.351182 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4zll\" (UniqueName: \"kubernetes.io/projected/18a49177-56d6-4bae-82c7-9007eeb56f15-kube-api-access-k4zll\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:31:28.351219 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.351213 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a49177-56d6-4bae-82c7-9007eeb56f15-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:31:28.351219 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.351226 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a49177-56d6-4bae-82c7-9007eeb56f15-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:31:28.538864 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.538821 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9"] Sep 08 14:31:28.544926 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:28.544899 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0416fe39-6f6b7f9976-phgf9"] Sep 08 14:31:30.243555 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:30.243519 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18a49177-56d6-4bae-82c7-9007eeb56f15" path="/var/lib/kubelet/pods/18a49177-56d6-4bae-82c7-9007eeb56f15/volumes" Sep 08 14:31:32.649988 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.649957 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx"] Sep 08 14:31:32.650418 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.650387 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18a49177-56d6-4bae-82c7-9007eeb56f15" containerName="maas-api" Sep 08 14:31:32.650418 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.650397 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="18a49177-56d6-4bae-82c7-9007eeb56f15" containerName="maas-api" Sep 08 14:31:32.650490 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.650460 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="18a49177-56d6-4bae-82c7-9007eeb56f15" containerName="maas-api" Sep 08 14:31:32.655048 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.655025 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:32.657149 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.657127 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5298415e-serving-cert\"" Sep 08 14:31:32.665718 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.665687 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx"] Sep 08 14:31:32.794927 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.794883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4db98a-7ad8-4132-8ad0-efdc3960073d-metrics-certs\") pod \"maas-api-e2e-ait-5298415e-69fdccc876-pmmpx\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:32.795116 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.794992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4db98a-7ad8-4132-8ad0-efdc3960073d-maas-api-tls\") pod \"maas-api-e2e-ait-5298415e-69fdccc876-pmmpx\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:32.795116 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.795054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fgcb\" (UniqueName: \"kubernetes.io/projected/9a4db98a-7ad8-4132-8ad0-efdc3960073d-kube-api-access-2fgcb\") pod \"maas-api-e2e-ait-5298415e-69fdccc876-pmmpx\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:32.795116 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.795098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4db98a-7ad8-4132-8ad0-efdc3960073d-openshift-service-ca\") pod \"maas-api-e2e-ait-5298415e-69fdccc876-pmmpx\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:32.896405 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.896366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4db98a-7ad8-4132-8ad0-efdc3960073d-maas-api-tls\") pod \"maas-api-e2e-ait-5298415e-69fdccc876-pmmpx\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:32.896601 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.896415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fgcb\" (UniqueName: \"kubernetes.io/projected/9a4db98a-7ad8-4132-8ad0-efdc3960073d-kube-api-access-2fgcb\") pod \"maas-api-e2e-ait-5298415e-69fdccc876-pmmpx\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:32.896601 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.896456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4db98a-7ad8-4132-8ad0-efdc3960073d-openshift-service-ca\") pod \"maas-api-e2e-ait-5298415e-69fdccc876-pmmpx\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:32.896601 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.896502 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4db98a-7ad8-4132-8ad0-efdc3960073d-metrics-certs\") pod \"maas-api-e2e-ait-5298415e-69fdccc876-pmmpx\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:32.897236 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.897210 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4db98a-7ad8-4132-8ad0-efdc3960073d-openshift-service-ca\") pod \"maas-api-e2e-ait-5298415e-69fdccc876-pmmpx\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:32.899040 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.899019 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4db98a-7ad8-4132-8ad0-efdc3960073d-maas-api-tls\") pod \"maas-api-e2e-ait-5298415e-69fdccc876-pmmpx\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:32.899146 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.899102 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4db98a-7ad8-4132-8ad0-efdc3960073d-metrics-certs\") pod \"maas-api-e2e-ait-5298415e-69fdccc876-pmmpx\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:32.904781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.904699 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fgcb\" (UniqueName: \"kubernetes.io/projected/9a4db98a-7ad8-4132-8ad0-efdc3960073d-kube-api-access-2fgcb\") pod \"maas-api-e2e-ait-5298415e-69fdccc876-pmmpx\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:32.969580 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:32.969542 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:33.245742 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:33.245697 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx"] Sep 08 14:31:33.248798 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:31:33.248432 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4db98a_7ad8_4132_8ad0_efdc3960073d.slice/crio-229faec6b2ad72634e47dfea723bc4cedbbfe43acfb92cfaf21ce6dd3dfe2faa WatchSource:0}: Error finding container 229faec6b2ad72634e47dfea723bc4cedbbfe43acfb92cfaf21ce6dd3dfe2faa: Status 404 returned error can't find the container with id 229faec6b2ad72634e47dfea723bc4cedbbfe43acfb92cfaf21ce6dd3dfe2faa Sep 08 14:31:34.266108 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:34.266071 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" event={"ID":"9a4db98a-7ad8-4132-8ad0-efdc3960073d","Type":"ContainerStarted","Data":"b2c5ae9a1699fa39e946f6396d153fef2f7d9d823525e1aa077afab72a81c6c0"} Sep 08 14:31:34.266527 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:34.266120 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" event={"ID":"9a4db98a-7ad8-4132-8ad0-efdc3960073d","Type":"ContainerStarted","Data":"229faec6b2ad72634e47dfea723bc4cedbbfe43acfb92cfaf21ce6dd3dfe2faa"} Sep 08 14:31:34.266527 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:34.266228 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:31:34.292347 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:34.292271 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" podStartSLOduration=1.9060785139999998 podStartE2EDuration="2.292254772s" totalImagesPullingTime="386.176258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:31:34.290308073 +0000 UTC m=+906.574625205" watchObservedRunningTime="2026-09-08 14:31:34.292254772 +0000 UTC m=+906.576571908" Sep 08 14:31:40.279467 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:31:40.279433 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:32:03.059558 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.059515 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx"] Sep 08 14:32:03.062362 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.059824 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" podUID="9a4db98a-7ad8-4132-8ad0-efdc3960073d" containerName="maas-api" containerID="cri-o://b2c5ae9a1699fa39e946f6396d153fef2f7d9d823525e1aa077afab72a81c6c0" gracePeriod=30 Sep 08 14:32:03.332426 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.332401 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:32:03.442934 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.442899 2578 generic.go:356] "Generic (PLEG): container finished" podID="9a4db98a-7ad8-4132-8ad0-efdc3960073d" containerID="b2c5ae9a1699fa39e946f6396d153fef2f7d9d823525e1aa077afab72a81c6c0" exitCode=0 Sep 08 14:32:03.443093 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.442946 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" Sep 08 14:32:03.443093 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.442982 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" event={"ID":"9a4db98a-7ad8-4132-8ad0-efdc3960073d","Type":"ContainerDied","Data":"b2c5ae9a1699fa39e946f6396d153fef2f7d9d823525e1aa077afab72a81c6c0"} Sep 08 14:32:03.443093 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.443052 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx" event={"ID":"9a4db98a-7ad8-4132-8ad0-efdc3960073d","Type":"ContainerDied","Data":"229faec6b2ad72634e47dfea723bc4cedbbfe43acfb92cfaf21ce6dd3dfe2faa"} Sep 08 14:32:03.443093 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.443083 2578 scope.go:117] "RemoveContainer" containerID="b2c5ae9a1699fa39e946f6396d153fef2f7d9d823525e1aa077afab72a81c6c0" Sep 08 14:32:03.456672 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.456645 2578 scope.go:117] "RemoveContainer" containerID="b2c5ae9a1699fa39e946f6396d153fef2f7d9d823525e1aa077afab72a81c6c0" Sep 08 14:32:03.457109 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:32:03.457079 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c5ae9a1699fa39e946f6396d153fef2f7d9d823525e1aa077afab72a81c6c0\": container with ID starting with b2c5ae9a1699fa39e946f6396d153fef2f7d9d823525e1aa077afab72a81c6c0 not found: ID does not exist" containerID="b2c5ae9a1699fa39e946f6396d153fef2f7d9d823525e1aa077afab72a81c6c0" Sep 08 14:32:03.457215 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.457120 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c5ae9a1699fa39e946f6396d153fef2f7d9d823525e1aa077afab72a81c6c0"} err="failed to get container status \"b2c5ae9a1699fa39e946f6396d153fef2f7d9d823525e1aa077afab72a81c6c0\": rpc error: code = NotFound desc = could not find container \"b2c5ae9a1699fa39e946f6396d153fef2f7d9d823525e1aa077afab72a81c6c0\": container with ID starting with b2c5ae9a1699fa39e946f6396d153fef2f7d9d823525e1aa077afab72a81c6c0 not found: ID does not exist" Sep 08 14:32:03.490483 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.490450 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fgcb\" (UniqueName: \"kubernetes.io/projected/9a4db98a-7ad8-4132-8ad0-efdc3960073d-kube-api-access-2fgcb\") pod \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " Sep 08 14:32:03.490635 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.490504 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4db98a-7ad8-4132-8ad0-efdc3960073d-openshift-service-ca\") pod \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " Sep 08 14:32:03.490635 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.490538 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4db98a-7ad8-4132-8ad0-efdc3960073d-maas-api-tls\") pod \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " Sep 08 14:32:03.490779 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.490683 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4db98a-7ad8-4132-8ad0-efdc3960073d-metrics-certs\") pod \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\" (UID: \"9a4db98a-7ad8-4132-8ad0-efdc3960073d\") " Sep 08 14:32:03.491137 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.490940 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a4db98a-7ad8-4132-8ad0-efdc3960073d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9a4db98a-7ad8-4132-8ad0-efdc3960073d" (UID: "9a4db98a-7ad8-4132-8ad0-efdc3960073d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:32:03.491137 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.491031 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a4db98a-7ad8-4132-8ad0-efdc3960073d-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:03.493121 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.493088 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a4db98a-7ad8-4132-8ad0-efdc3960073d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9a4db98a-7ad8-4132-8ad0-efdc3960073d" (UID: "9a4db98a-7ad8-4132-8ad0-efdc3960073d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:32:03.493350 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.493326 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4db98a-7ad8-4132-8ad0-efdc3960073d-kube-api-access-2fgcb" (OuterVolumeSpecName: "kube-api-access-2fgcb") pod "9a4db98a-7ad8-4132-8ad0-efdc3960073d" (UID: "9a4db98a-7ad8-4132-8ad0-efdc3960073d"). InnerVolumeSpecName "kube-api-access-2fgcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:32:03.493504 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.493478 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a4db98a-7ad8-4132-8ad0-efdc3960073d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9a4db98a-7ad8-4132-8ad0-efdc3960073d" (UID: "9a4db98a-7ad8-4132-8ad0-efdc3960073d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:32:03.591963 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.591870 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9a4db98a-7ad8-4132-8ad0-efdc3960073d-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:03.591963 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.591901 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fgcb\" (UniqueName: \"kubernetes.io/projected/9a4db98a-7ad8-4132-8ad0-efdc3960073d-kube-api-access-2fgcb\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:03.591963 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.591914 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9a4db98a-7ad8-4132-8ad0-efdc3960073d-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:03.766046 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.766017 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx"] Sep 08 14:32:03.772177 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:03.772153 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5298415e-69fdccc876-pmmpx"] Sep 08 14:32:04.242479 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:04.242449 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a4db98a-7ad8-4132-8ad0-efdc3960073d" path="/var/lib/kubelet/pods/9a4db98a-7ad8-4132-8ad0-efdc3960073d/volumes" Sep 08 14:32:08.938951 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:08.938913 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9"] Sep 08 14:32:08.939572 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:08.939551 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a4db98a-7ad8-4132-8ad0-efdc3960073d" containerName="maas-api" Sep 08 14:32:08.939678 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:08.939577 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a4db98a-7ad8-4132-8ad0-efdc3960073d" containerName="maas-api" Sep 08 14:32:08.939753 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:08.939679 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a4db98a-7ad8-4132-8ad0-efdc3960073d" containerName="maas-api" Sep 08 14:32:08.944474 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:08.944447 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:08.946846 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:08.946822 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-c7c09416-serving-cert\"" Sep 08 14:32:08.956944 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:08.956913 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9"] Sep 08 14:32:09.044888 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.044850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-maas-api-tls\") pod \"maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:09.045065 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.044936 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-metrics-certs\") pod \"maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:09.045065 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.044969 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f26772-0d43-4751-824d-4969e2f7043f-openshift-service-ca\") pod \"maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:09.045065 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.044990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tsfn\" (UniqueName: \"kubernetes.io/projected/f7f26772-0d43-4751-824d-4969e2f7043f-kube-api-access-9tsfn\") pod \"maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:09.146304 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.146273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-maas-api-tls\") pod \"maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:09.146507 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.146326 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-metrics-certs\") pod \"maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:09.146507 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.146350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f26772-0d43-4751-824d-4969e2f7043f-openshift-service-ca\") pod \"maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:09.146507 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.146367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tsfn\" (UniqueName: \"kubernetes.io/projected/f7f26772-0d43-4751-824d-4969e2f7043f-kube-api-access-9tsfn\") pod \"maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:09.146507 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:32:09.146431 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-serving-cert: secret "maas-api-e2e-derive-c7c09416-serving-cert" not found Sep 08 14:32:09.146717 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:32:09.146520 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-maas-api-tls podName:f7f26772-0d43-4751-824d-4969e2f7043f nodeName:}" failed. No retries permitted until 2026-09-08 14:32:09.646496721 +0000 UTC m=+941.930813813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-maas-api-tls") pod "maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" (UID: "f7f26772-0d43-4751-824d-4969e2f7043f") : secret "maas-api-e2e-derive-c7c09416-serving-cert" not found Sep 08 14:32:09.147123 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.147105 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f26772-0d43-4751-824d-4969e2f7043f-openshift-service-ca\") pod \"maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:09.148949 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.148933 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-metrics-certs\") pod \"maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:09.155155 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.155131 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tsfn\" (UniqueName: \"kubernetes.io/projected/f7f26772-0d43-4751-824d-4969e2f7043f-kube-api-access-9tsfn\") pod \"maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:09.651756 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.651710 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-maas-api-tls\") pod \"maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:09.654382 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.654353 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-maas-api-tls\") pod \"maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:09.859897 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:09.859851 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:10.137184 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:10.137158 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9"] Sep 08 14:32:10.138282 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:32:10.138255 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f26772_0d43_4751_824d_4969e2f7043f.slice/crio-c0e66d91dc7135000a55bcc9b564e16398f03bd626c429840938428f25042775 WatchSource:0}: Error finding container c0e66d91dc7135000a55bcc9b564e16398f03bd626c429840938428f25042775: Status 404 returned error can't find the container with id c0e66d91dc7135000a55bcc9b564e16398f03bd626c429840938428f25042775 Sep 08 14:32:10.486662 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:10.486625 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" event={"ID":"f7f26772-0d43-4751-824d-4969e2f7043f","Type":"ContainerStarted","Data":"c0e66d91dc7135000a55bcc9b564e16398f03bd626c429840938428f25042775"} Sep 08 14:32:11.494269 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:11.494232 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" event={"ID":"f7f26772-0d43-4751-824d-4969e2f7043f","Type":"ContainerStarted","Data":"de17b973c86a9f2963e601ba75066ac84da732233d7b8ed8d73816610eebb74f"} Sep 08 14:32:11.494656 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:11.494291 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:11.511986 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:11.511939 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" podStartSLOduration=3.137022377 podStartE2EDuration="3.511925324s" totalImagesPullingTime="374.902947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:32:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:32:11.510086614 +0000 UTC m=+943.794403728" watchObservedRunningTime="2026-09-08 14:32:11.511925324 +0000 UTC m=+943.796242438" Sep 08 14:32:17.506215 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:17.506184 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:35.320943 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.320906 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl"] Sep 08 14:32:35.321560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.321158 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" podUID="d03b0eee-8606-4423-9a4c-5e944be993b2" containerName="maas-api" containerID="cri-o://ecec1c1e975b4ba33cc187e798cc3ae0ada70524294a0528abef24f977ea75b2" gracePeriod=30 Sep 08 14:32:35.380194 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.380156 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6"] Sep 08 14:32:35.380455 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.380421 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" podUID="bea56ce4-2712-466f-86d2-736ca9eb3956" containerName="payload-processing" containerID="cri-o://ed2efaa036162e3075f78ae7f9744e533c54e63ccafb7eff99a3da8379e38afc" gracePeriod=30 Sep 08 14:32:35.643776 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.643718 2578 generic.go:356] "Generic (PLEG): container finished" podID="bea56ce4-2712-466f-86d2-736ca9eb3956" containerID="ed2efaa036162e3075f78ae7f9744e533c54e63ccafb7eff99a3da8379e38afc" exitCode=0 Sep 08 14:32:35.643776 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.643767 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" event={"ID":"bea56ce4-2712-466f-86d2-736ca9eb3956","Type":"ContainerDied","Data":"ed2efaa036162e3075f78ae7f9744e533c54e63ccafb7eff99a3da8379e38afc"} Sep 08 14:32:35.645414 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.645391 2578 generic.go:356] "Generic (PLEG): container finished" podID="d03b0eee-8606-4423-9a4c-5e944be993b2" containerID="ecec1c1e975b4ba33cc187e798cc3ae0ada70524294a0528abef24f977ea75b2" exitCode=0 Sep 08 14:32:35.645536 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.645423 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" event={"ID":"d03b0eee-8606-4423-9a4c-5e944be993b2","Type":"ContainerDied","Data":"ecec1c1e975b4ba33cc187e798cc3ae0ada70524294a0528abef24f977ea75b2"} Sep 08 14:32:35.677380 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.677348 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:32:35.680429 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.680407 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" Sep 08 14:32:35.806600 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.806569 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-metrics-certs\") pod \"d03b0eee-8606-4423-9a4c-5e944be993b2\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " Sep 08 14:32:35.806833 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.806620 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-maas-api-tls\") pod \"d03b0eee-8606-4423-9a4c-5e944be993b2\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " Sep 08 14:32:35.806833 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.806648 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6mqn\" (UniqueName: \"kubernetes.io/projected/d03b0eee-8606-4423-9a4c-5e944be993b2-kube-api-access-m6mqn\") pod \"d03b0eee-8606-4423-9a4c-5e944be993b2\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " Sep 08 14:32:35.806833 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.806774 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03b0eee-8606-4423-9a4c-5e944be993b2-openshift-service-ca\") pod \"d03b0eee-8606-4423-9a4c-5e944be993b2\" (UID: \"d03b0eee-8606-4423-9a4c-5e944be993b2\") " Sep 08 14:32:35.806997 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.806839 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea56ce4-2712-466f-86d2-736ca9eb3956-plugins-config-volume\") pod \"bea56ce4-2712-466f-86d2-736ca9eb3956\" (UID: \"bea56ce4-2712-466f-86d2-736ca9eb3956\") " Sep 08 14:32:35.806997 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.806921 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thlfs\" (UniqueName: \"kubernetes.io/projected/bea56ce4-2712-466f-86d2-736ca9eb3956-kube-api-access-thlfs\") pod \"bea56ce4-2712-466f-86d2-736ca9eb3956\" (UID: \"bea56ce4-2712-466f-86d2-736ca9eb3956\") " Sep 08 14:32:35.807153 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.807125 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03b0eee-8606-4423-9a4c-5e944be993b2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d03b0eee-8606-4423-9a4c-5e944be993b2" (UID: "d03b0eee-8606-4423-9a4c-5e944be993b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:32:35.807265 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.807206 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea56ce4-2712-466f-86d2-736ca9eb3956-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bea56ce4-2712-466f-86d2-736ca9eb3956" (UID: "bea56ce4-2712-466f-86d2-736ca9eb3956"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:32:35.807327 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.807267 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03b0eee-8606-4423-9a4c-5e944be993b2-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:35.809109 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.809084 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d03b0eee-8606-4423-9a4c-5e944be993b2" (UID: "d03b0eee-8606-4423-9a4c-5e944be993b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:32:35.809109 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.809098 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea56ce4-2712-466f-86d2-736ca9eb3956-kube-api-access-thlfs" (OuterVolumeSpecName: "kube-api-access-thlfs") pod "bea56ce4-2712-466f-86d2-736ca9eb3956" (UID: "bea56ce4-2712-466f-86d2-736ca9eb3956"). InnerVolumeSpecName "kube-api-access-thlfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:32:35.809345 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.809328 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03b0eee-8606-4423-9a4c-5e944be993b2-kube-api-access-m6mqn" (OuterVolumeSpecName: "kube-api-access-m6mqn") pod "d03b0eee-8606-4423-9a4c-5e944be993b2" (UID: "d03b0eee-8606-4423-9a4c-5e944be993b2"). InnerVolumeSpecName "kube-api-access-m6mqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:32:35.809395 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.809356 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d03b0eee-8606-4423-9a4c-5e944be993b2" (UID: "d03b0eee-8606-4423-9a4c-5e944be993b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:32:35.908206 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.908120 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:35.908206 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.908148 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6mqn\" (UniqueName: \"kubernetes.io/projected/d03b0eee-8606-4423-9a4c-5e944be993b2-kube-api-access-m6mqn\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:35.908206 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.908159 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bea56ce4-2712-466f-86d2-736ca9eb3956-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:35.908206 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.908169 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thlfs\" (UniqueName: \"kubernetes.io/projected/bea56ce4-2712-466f-86d2-736ca9eb3956-kube-api-access-thlfs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:35.908206 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:35.908178 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d03b0eee-8606-4423-9a4c-5e944be993b2-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:36.652610 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:36.652515 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" event={"ID":"bea56ce4-2712-466f-86d2-736ca9eb3956","Type":"ContainerDied","Data":"871650fa16d1ca02d8e672d734a80c08bf250eeed7f97bc2935ac3ba49d15fb0"} Sep 08 14:32:36.652610 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:36.652589 2578 scope.go:117] "RemoveContainer" containerID="ed2efaa036162e3075f78ae7f9744e533c54e63ccafb7eff99a3da8379e38afc" Sep 08 14:32:36.653121 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:36.652529 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6" Sep 08 14:32:36.653974 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:36.653960 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" event={"ID":"d03b0eee-8606-4423-9a4c-5e944be993b2","Type":"ContainerDied","Data":"c95abe2ae04b0d028e502bb8a83ea0ff8a1e5b1da114d41a84b55fa6ddb33f44"} Sep 08 14:32:36.654044 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:36.653991 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl" Sep 08 14:32:36.662058 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:36.662042 2578 scope.go:117] "RemoveContainer" containerID="ecec1c1e975b4ba33cc187e798cc3ae0ada70524294a0528abef24f977ea75b2" Sep 08 14:32:36.673537 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:36.673513 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl"] Sep 08 14:32:36.679394 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:36.679371 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-47d144-546f6bcc4b-rngcl"] Sep 08 14:32:36.690353 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:36.690323 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6"] Sep 08 14:32:36.694374 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:36.694352 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-47d144-bf58f96d5-gztf6"] Sep 08 14:32:38.242241 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:38.242208 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bea56ce4-2712-466f-86d2-736ca9eb3956" path="/var/lib/kubelet/pods/bea56ce4-2712-466f-86d2-736ca9eb3956/volumes" Sep 08 14:32:38.242611 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:38.242586 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d03b0eee-8606-4423-9a4c-5e944be993b2" path="/var/lib/kubelet/pods/d03b0eee-8606-4423-9a4c-5e944be993b2/volumes" Sep 08 14:32:39.893760 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:39.893259 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9"] Sep 08 14:32:39.893760 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:39.893498 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" podUID="f7f26772-0d43-4751-824d-4969e2f7043f" containerName="maas-api" containerID="cri-o://de17b973c86a9f2963e601ba75066ac84da732233d7b8ed8d73816610eebb74f" gracePeriod=30 Sep 08 14:32:40.156602 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.156537 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:40.247426 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.247389 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-maas-api-tls\") pod \"f7f26772-0d43-4751-824d-4969e2f7043f\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " Sep 08 14:32:40.247594 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.247547 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f26772-0d43-4751-824d-4969e2f7043f-openshift-service-ca\") pod \"f7f26772-0d43-4751-824d-4969e2f7043f\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " Sep 08 14:32:40.247594 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.247587 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tsfn\" (UniqueName: \"kubernetes.io/projected/f7f26772-0d43-4751-824d-4969e2f7043f-kube-api-access-9tsfn\") pod \"f7f26772-0d43-4751-824d-4969e2f7043f\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " Sep 08 14:32:40.247757 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.247636 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-metrics-certs\") pod \"f7f26772-0d43-4751-824d-4969e2f7043f\" (UID: \"f7f26772-0d43-4751-824d-4969e2f7043f\") " Sep 08 14:32:40.249151 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.249118 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f26772-0d43-4751-824d-4969e2f7043f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f7f26772-0d43-4751-824d-4969e2f7043f" (UID: "f7f26772-0d43-4751-824d-4969e2f7043f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:32:40.251461 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.251431 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f7f26772-0d43-4751-824d-4969e2f7043f" (UID: "f7f26772-0d43-4751-824d-4969e2f7043f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:32:40.252929 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.252896 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f7f26772-0d43-4751-824d-4969e2f7043f" (UID: "f7f26772-0d43-4751-824d-4969e2f7043f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:32:40.255757 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.255705 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f26772-0d43-4751-824d-4969e2f7043f-kube-api-access-9tsfn" (OuterVolumeSpecName: "kube-api-access-9tsfn") pod "f7f26772-0d43-4751-824d-4969e2f7043f" (UID: "f7f26772-0d43-4751-824d-4969e2f7043f"). InnerVolumeSpecName "kube-api-access-9tsfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:32:40.348606 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.348565 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7f26772-0d43-4751-824d-4969e2f7043f-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:40.348606 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.348597 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tsfn\" (UniqueName: \"kubernetes.io/projected/f7f26772-0d43-4751-824d-4969e2f7043f-kube-api-access-9tsfn\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:40.348606 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.348607 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:40.348606 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.348615 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7f26772-0d43-4751-824d-4969e2f7043f-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:32:40.682878 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.682841 2578 generic.go:356] "Generic (PLEG): container finished" podID="f7f26772-0d43-4751-824d-4969e2f7043f" containerID="de17b973c86a9f2963e601ba75066ac84da732233d7b8ed8d73816610eebb74f" exitCode=0 Sep 08 14:32:40.683045 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.682890 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" Sep 08 14:32:40.683045 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.682927 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" event={"ID":"f7f26772-0d43-4751-824d-4969e2f7043f","Type":"ContainerDied","Data":"de17b973c86a9f2963e601ba75066ac84da732233d7b8ed8d73816610eebb74f"} Sep 08 14:32:40.683045 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.682981 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9" event={"ID":"f7f26772-0d43-4751-824d-4969e2f7043f","Type":"ContainerDied","Data":"c0e66d91dc7135000a55bcc9b564e16398f03bd626c429840938428f25042775"} Sep 08 14:32:40.683045 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.683006 2578 scope.go:117] "RemoveContainer" containerID="de17b973c86a9f2963e601ba75066ac84da732233d7b8ed8d73816610eebb74f" Sep 08 14:32:40.693286 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.693270 2578 scope.go:117] "RemoveContainer" containerID="de17b973c86a9f2963e601ba75066ac84da732233d7b8ed8d73816610eebb74f" Sep 08 14:32:40.693529 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:32:40.693511 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de17b973c86a9f2963e601ba75066ac84da732233d7b8ed8d73816610eebb74f\": container with ID starting with de17b973c86a9f2963e601ba75066ac84da732233d7b8ed8d73816610eebb74f not found: ID does not exist" containerID="de17b973c86a9f2963e601ba75066ac84da732233d7b8ed8d73816610eebb74f" Sep 08 14:32:40.693585 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.693539 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de17b973c86a9f2963e601ba75066ac84da732233d7b8ed8d73816610eebb74f"} err="failed to get container status \"de17b973c86a9f2963e601ba75066ac84da732233d7b8ed8d73816610eebb74f\": rpc error: code = NotFound desc = could not find container \"de17b973c86a9f2963e601ba75066ac84da732233d7b8ed8d73816610eebb74f\": container with ID starting with de17b973c86a9f2963e601ba75066ac84da732233d7b8ed8d73816610eebb74f not found: ID does not exist" Sep 08 14:32:40.705087 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.705053 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9"] Sep 08 14:32:40.709246 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:40.709218 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-c7c09416-6bbf5fbbb5-w7gj9"] Sep 08 14:32:42.241924 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:42.241891 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7f26772-0d43-4751-824d-4969e2f7043f" path="/var/lib/kubelet/pods/f7f26772-0d43-4751-824d-4969e2f7043f/volumes" Sep 08 14:32:50.939713 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:50.939611 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l"] Sep 08 14:32:50.940318 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:50.940296 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bea56ce4-2712-466f-86d2-736ca9eb3956" containerName="payload-processing" Sep 08 14:32:50.940360 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:50.940323 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bea56ce4-2712-466f-86d2-736ca9eb3956" containerName="payload-processing" Sep 08 14:32:50.940360 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:50.940350 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d03b0eee-8606-4423-9a4c-5e944be993b2" containerName="maas-api" Sep 08 14:32:50.940420 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:50.940360 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d03b0eee-8606-4423-9a4c-5e944be993b2" containerName="maas-api" Sep 08 14:32:50.940420 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:50.940379 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f26772-0d43-4751-824d-4969e2f7043f" containerName="maas-api" Sep 08 14:32:50.940420 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:50.940388 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f26772-0d43-4751-824d-4969e2f7043f" containerName="maas-api" Sep 08 14:32:50.940518 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:50.940499 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d03b0eee-8606-4423-9a4c-5e944be993b2" containerName="maas-api" Sep 08 14:32:50.940554 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:50.940518 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bea56ce4-2712-466f-86d2-736ca9eb3956" containerName="payload-processing" Sep 08 14:32:50.940554 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:50.940529 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f26772-0d43-4751-824d-4969e2f7043f" containerName="maas-api" Sep 08 14:32:50.944435 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:50.944416 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:50.946563 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:50.946537 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-ae4649-serving-cert\"" Sep 08 14:32:50.954652 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:50.954617 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l"] Sep 08 14:32:51.056906 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.056864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmf5r\" (UniqueName: \"kubernetes.io/projected/61ccf067-3418-4374-a7f3-19aed780aa3d-kube-api-access-mmf5r\") pod \"maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:51.056906 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.056922 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ccf067-3418-4374-a7f3-19aed780aa3d-openshift-service-ca\") pod \"maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:51.057166 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.057022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-maas-api-tls\") pod \"maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:51.057166 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.057054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-metrics-certs\") pod \"maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:51.158302 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.158266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-maas-api-tls\") pod \"maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:51.158302 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.158299 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-metrics-certs\") pod \"maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:51.158529 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.158372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmf5r\" (UniqueName: \"kubernetes.io/projected/61ccf067-3418-4374-a7f3-19aed780aa3d-kube-api-access-mmf5r\") pod \"maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:51.158529 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.158392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ccf067-3418-4374-a7f3-19aed780aa3d-openshift-service-ca\") pod \"maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:51.158529 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:32:51.158418 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-serving-cert: secret "maas-api-e2e-explicit-ae4649-serving-cert" not found Sep 08 14:32:51.158529 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:32:51.158501 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-maas-api-tls podName:61ccf067-3418-4374-a7f3-19aed780aa3d nodeName:}" failed. No retries permitted until 2026-09-08 14:32:51.658483587 +0000 UTC m=+983.942800700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-maas-api-tls") pod "maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" (UID: "61ccf067-3418-4374-a7f3-19aed780aa3d") : secret "maas-api-e2e-explicit-ae4649-serving-cert" not found Sep 08 14:32:51.159017 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.159000 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ccf067-3418-4374-a7f3-19aed780aa3d-openshift-service-ca\") pod \"maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:51.161078 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.161055 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-metrics-certs\") pod \"maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:51.166792 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.166763 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmf5r\" (UniqueName: \"kubernetes.io/projected/61ccf067-3418-4374-a7f3-19aed780aa3d-kube-api-access-mmf5r\") pod \"maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:51.663118 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.663085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-maas-api-tls\") pod \"maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:51.665635 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.665609 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-maas-api-tls\") pod \"maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:51.857747 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:51.857687 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:52.137856 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:52.137825 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l"] Sep 08 14:32:52.139045 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:32:52.139019 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ccf067_3418_4374_a7f3_19aed780aa3d.slice/crio-a30f5357685807c40bd3798f58636502e8529d622330541f62b29938d5f97d9d WatchSource:0}: Error finding container a30f5357685807c40bd3798f58636502e8529d622330541f62b29938d5f97d9d: Status 404 returned error can't find the container with id a30f5357685807c40bd3798f58636502e8529d622330541f62b29938d5f97d9d Sep 08 14:32:52.140466 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:52.140450 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 14:32:52.763344 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:52.763305 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" event={"ID":"61ccf067-3418-4374-a7f3-19aed780aa3d","Type":"ContainerStarted","Data":"01540ebbd67945a8e7c23edf27df445823f5807d89350ae940e52d5f69fbfd49"} Sep 08 14:32:52.763556 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:52.763383 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" event={"ID":"61ccf067-3418-4374-a7f3-19aed780aa3d","Type":"ContainerStarted","Data":"a30f5357685807c40bd3798f58636502e8529d622330541f62b29938d5f97d9d"} Sep 08 14:32:52.763556 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:52.763539 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:32:52.782443 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:52.782384 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" podStartSLOduration=2.378751694 podStartE2EDuration="2.78236184s" totalImagesPullingTime="403.610146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:32:52.779381013 +0000 UTC m=+985.063698135" watchObservedRunningTime="2026-09-08 14:32:52.78236184 +0000 UTC m=+985.066678957" Sep 08 14:32:58.775399 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:32:58.775368 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:34:03.792816 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:03.792776 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l"] Sep 08 14:34:03.793308 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:03.792974 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" podUID="61ccf067-3418-4374-a7f3-19aed780aa3d" containerName="maas-api" containerID="cri-o://01540ebbd67945a8e7c23edf27df445823f5807d89350ae940e52d5f69fbfd49" gracePeriod=30 Sep 08 14:34:04.047269 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.047196 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:34:04.163186 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.163156 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-metrics-certs\") pod \"61ccf067-3418-4374-a7f3-19aed780aa3d\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " Sep 08 14:34:04.163350 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.163198 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-maas-api-tls\") pod \"61ccf067-3418-4374-a7f3-19aed780aa3d\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " Sep 08 14:34:04.163350 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.163266 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ccf067-3418-4374-a7f3-19aed780aa3d-openshift-service-ca\") pod \"61ccf067-3418-4374-a7f3-19aed780aa3d\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " Sep 08 14:34:04.163350 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.163341 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmf5r\" (UniqueName: \"kubernetes.io/projected/61ccf067-3418-4374-a7f3-19aed780aa3d-kube-api-access-mmf5r\") pod \"61ccf067-3418-4374-a7f3-19aed780aa3d\" (UID: \"61ccf067-3418-4374-a7f3-19aed780aa3d\") " Sep 08 14:34:04.163719 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.163687 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ccf067-3418-4374-a7f3-19aed780aa3d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "61ccf067-3418-4374-a7f3-19aed780aa3d" (UID: "61ccf067-3418-4374-a7f3-19aed780aa3d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:34:04.165561 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.165537 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "61ccf067-3418-4374-a7f3-19aed780aa3d" (UID: "61ccf067-3418-4374-a7f3-19aed780aa3d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:34:04.165655 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.165581 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "61ccf067-3418-4374-a7f3-19aed780aa3d" (UID: "61ccf067-3418-4374-a7f3-19aed780aa3d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:34:04.165655 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.165610 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ccf067-3418-4374-a7f3-19aed780aa3d-kube-api-access-mmf5r" (OuterVolumeSpecName: "kube-api-access-mmf5r") pod "61ccf067-3418-4374-a7f3-19aed780aa3d" (UID: "61ccf067-3418-4374-a7f3-19aed780aa3d"). InnerVolumeSpecName "kube-api-access-mmf5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:34:04.179210 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.179170 2578 generic.go:356] "Generic (PLEG): container finished" podID="61ccf067-3418-4374-a7f3-19aed780aa3d" containerID="01540ebbd67945a8e7c23edf27df445823f5807d89350ae940e52d5f69fbfd49" exitCode=0 Sep 08 14:34:04.179326 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.179215 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" Sep 08 14:34:04.179326 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.179258 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" event={"ID":"61ccf067-3418-4374-a7f3-19aed780aa3d","Type":"ContainerDied","Data":"01540ebbd67945a8e7c23edf27df445823f5807d89350ae940e52d5f69fbfd49"} Sep 08 14:34:04.179437 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.179328 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l" event={"ID":"61ccf067-3418-4374-a7f3-19aed780aa3d","Type":"ContainerDied","Data":"a30f5357685807c40bd3798f58636502e8529d622330541f62b29938d5f97d9d"} Sep 08 14:34:04.179437 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.179359 2578 scope.go:117] "RemoveContainer" containerID="01540ebbd67945a8e7c23edf27df445823f5807d89350ae940e52d5f69fbfd49" Sep 08 14:34:04.191466 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.191444 2578 scope.go:117] "RemoveContainer" containerID="01540ebbd67945a8e7c23edf27df445823f5807d89350ae940e52d5f69fbfd49" Sep 08 14:34:04.191807 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:34:04.191779 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01540ebbd67945a8e7c23edf27df445823f5807d89350ae940e52d5f69fbfd49\": container with ID starting with 01540ebbd67945a8e7c23edf27df445823f5807d89350ae940e52d5f69fbfd49 not found: ID does not exist" containerID="01540ebbd67945a8e7c23edf27df445823f5807d89350ae940e52d5f69fbfd49" Sep 08 14:34:04.191902 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.191815 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01540ebbd67945a8e7c23edf27df445823f5807d89350ae940e52d5f69fbfd49"} err="failed to get container status \"01540ebbd67945a8e7c23edf27df445823f5807d89350ae940e52d5f69fbfd49\": rpc error: code = NotFound desc = could not find container \"01540ebbd67945a8e7c23edf27df445823f5807d89350ae940e52d5f69fbfd49\": container with ID starting with 01540ebbd67945a8e7c23edf27df445823f5807d89350ae940e52d5f69fbfd49 not found: ID does not exist" Sep 08 14:34:04.203856 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.203828 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l"] Sep 08 14:34:04.210617 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.210595 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ae4649-76cb87bdd6-tmf9l"] Sep 08 14:34:04.242026 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.241984 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61ccf067-3418-4374-a7f3-19aed780aa3d" path="/var/lib/kubelet/pods/61ccf067-3418-4374-a7f3-19aed780aa3d/volumes" Sep 08 14:34:04.264155 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.264130 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmf5r\" (UniqueName: \"kubernetes.io/projected/61ccf067-3418-4374-a7f3-19aed780aa3d-kube-api-access-mmf5r\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:34:04.264155 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.264153 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:34:04.264281 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.264164 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/61ccf067-3418-4374-a7f3-19aed780aa3d-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:34:04.264281 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:04.264172 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/61ccf067-3418-4374-a7f3-19aed780aa3d-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:34:23.636784 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.636674 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx"] Sep 08 14:34:23.637193 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.637176 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61ccf067-3418-4374-a7f3-19aed780aa3d" containerName="maas-api" Sep 08 14:34:23.637243 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.637195 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="61ccf067-3418-4374-a7f3-19aed780aa3d" containerName="maas-api" Sep 08 14:34:23.637291 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.637282 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="61ccf067-3418-4374-a7f3-19aed780aa3d" containerName="maas-api" Sep 08 14:34:23.640348 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.640329 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:23.642528 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.642509 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a8fb2c38-serving-cert\"" Sep 08 14:34:23.653039 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.653017 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx"] Sep 08 14:34:23.748941 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.748908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6615426c-0495-4fc5-a631-75db3c59336a-openshift-service-ca\") pod \"maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:23.749088 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.748948 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krjd8\" (UniqueName: \"kubernetes.io/projected/6615426c-0495-4fc5-a631-75db3c59336a-kube-api-access-krjd8\") pod \"maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:23.749088 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.749033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6615426c-0495-4fc5-a631-75db3c59336a-maas-api-tls\") pod \"maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:23.749088 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.749079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6615426c-0495-4fc5-a631-75db3c59336a-metrics-certs\") pod \"maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:23.849638 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.849609 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krjd8\" (UniqueName: \"kubernetes.io/projected/6615426c-0495-4fc5-a631-75db3c59336a-kube-api-access-krjd8\") pod \"maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:23.849856 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.849689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6615426c-0495-4fc5-a631-75db3c59336a-maas-api-tls\") pod \"maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:23.849856 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.849800 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6615426c-0495-4fc5-a631-75db3c59336a-metrics-certs\") pod \"maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:23.850014 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.849882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6615426c-0495-4fc5-a631-75db3c59336a-openshift-service-ca\") pod \"maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:23.850523 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.850454 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6615426c-0495-4fc5-a631-75db3c59336a-openshift-service-ca\") pod \"maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:23.852353 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.852325 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6615426c-0495-4fc5-a631-75db3c59336a-metrics-certs\") pod \"maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:23.852353 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.852341 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6615426c-0495-4fc5-a631-75db3c59336a-maas-api-tls\") pod \"maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:23.860525 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.860489 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krjd8\" (UniqueName: \"kubernetes.io/projected/6615426c-0495-4fc5-a631-75db3c59336a-kube-api-access-krjd8\") pod \"maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:23.952816 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:23.952712 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:24.228963 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:24.228936 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx"] Sep 08 14:34:24.231744 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:34:24.231700 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6615426c_0495_4fc5_a631_75db3c59336a.slice/crio-ec1356aec544e8c4466ebef0999ef8c29e855dd9b8fec5ba143d5c6d2eac76b3 WatchSource:0}: Error finding container ec1356aec544e8c4466ebef0999ef8c29e855dd9b8fec5ba143d5c6d2eac76b3: Status 404 returned error can't find the container with id ec1356aec544e8c4466ebef0999ef8c29e855dd9b8fec5ba143d5c6d2eac76b3 Sep 08 14:34:24.295215 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:24.295178 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" event={"ID":"6615426c-0495-4fc5-a631-75db3c59336a","Type":"ContainerStarted","Data":"ec1356aec544e8c4466ebef0999ef8c29e855dd9b8fec5ba143d5c6d2eac76b3"} Sep 08 14:34:25.302623 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:25.302591 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" event={"ID":"6615426c-0495-4fc5-a631-75db3c59336a","Type":"ContainerStarted","Data":"42c8177678609ceed5fd1dcc58464692f58f177d84387e01307211b158bb2dae"} Sep 08 14:34:25.303018 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:25.302671 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:25.319633 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:25.319585 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" podStartSLOduration=1.7785383609999998 podStartE2EDuration="2.319570867s" totalImagesPullingTime="541.032506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:34:25.317949522 +0000 UTC m=+1077.602266651" watchObservedRunningTime="2026-09-08 14:34:25.319570867 +0000 UTC m=+1077.603887982" Sep 08 14:34:31.315607 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:31.315571 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:34:59.200262 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.200227 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm"] Sep 08 14:34:59.201009 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.200484 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" podUID="07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d" containerName="maas-api" containerID="cri-o://96e236a9ea01bf9291ccd02573f55e8d31bf35f0c0709e5b3bbde2799361b7bf" gracePeriod=30 Sep 08 14:34:59.273512 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.273477 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h"] Sep 08 14:34:59.273802 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.273768 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" podUID="bf5f5318-9253-407d-adfc-ea741d4e26ca" containerName="payload-processing" containerID="cri-o://fbb5f8bb4729c203b0233d9635a7f7e80bda45e58b95b62e3f096ae2f3bce910" gracePeriod=30 Sep 08 14:34:59.277967 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.277825 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2"] Sep 08 14:34:59.278219 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.278158 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" podUID="9ee38e92-c346-4bd7-9555-b8ec8253f10a" containerName="payload-pre-processing" containerID="cri-o://091fe0d1a62a946f4f189cad9ba769b3dc9de0b4ff702fcd98aeac72a6bd3b6e" gracePeriod=30 Sep 08 14:34:59.499979 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.499949 2578 generic.go:356] "Generic (PLEG): container finished" podID="bf5f5318-9253-407d-adfc-ea741d4e26ca" containerID="fbb5f8bb4729c203b0233d9635a7f7e80bda45e58b95b62e3f096ae2f3bce910" exitCode=0 Sep 08 14:34:59.499979 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.499988 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" event={"ID":"bf5f5318-9253-407d-adfc-ea741d4e26ca","Type":"ContainerDied","Data":"fbb5f8bb4729c203b0233d9635a7f7e80bda45e58b95b62e3f096ae2f3bce910"} Sep 08 14:34:59.502309 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.502289 2578 generic.go:356] "Generic (PLEG): container finished" podID="9ee38e92-c346-4bd7-9555-b8ec8253f10a" containerID="091fe0d1a62a946f4f189cad9ba769b3dc9de0b4ff702fcd98aeac72a6bd3b6e" exitCode=0 Sep 08 14:34:59.502437 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.502366 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" event={"ID":"9ee38e92-c346-4bd7-9555-b8ec8253f10a","Type":"ContainerDied","Data":"091fe0d1a62a946f4f189cad9ba769b3dc9de0b4ff702fcd98aeac72a6bd3b6e"} Sep 08 14:34:59.572821 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.572782 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" Sep 08 14:34:59.575856 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.575832 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" Sep 08 14:34:59.653186 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.653166 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:34:59.674650 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.674624 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kldfv\" (UniqueName: \"kubernetes.io/projected/9ee38e92-c346-4bd7-9555-b8ec8253f10a-kube-api-access-kldfv\") pod \"9ee38e92-c346-4bd7-9555-b8ec8253f10a\" (UID: \"9ee38e92-c346-4bd7-9555-b8ec8253f10a\") " Sep 08 14:34:59.674817 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.674707 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5f5318-9253-407d-adfc-ea741d4e26ca-plugins-config-volume\") pod \"bf5f5318-9253-407d-adfc-ea741d4e26ca\" (UID: \"bf5f5318-9253-407d-adfc-ea741d4e26ca\") " Sep 08 14:34:59.674817 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.674769 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9v4x\" (UniqueName: \"kubernetes.io/projected/bf5f5318-9253-407d-adfc-ea741d4e26ca-kube-api-access-x9v4x\") pod \"bf5f5318-9253-407d-adfc-ea741d4e26ca\" (UID: \"bf5f5318-9253-407d-adfc-ea741d4e26ca\") " Sep 08 14:34:59.674939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.674820 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee38e92-c346-4bd7-9555-b8ec8253f10a-plugins-config-volume\") pod \"9ee38e92-c346-4bd7-9555-b8ec8253f10a\" (UID: \"9ee38e92-c346-4bd7-9555-b8ec8253f10a\") " Sep 08 14:34:59.675202 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.675165 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf5f5318-9253-407d-adfc-ea741d4e26ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf5f5318-9253-407d-adfc-ea741d4e26ca" (UID: "bf5f5318-9253-407d-adfc-ea741d4e26ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:34:59.675301 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.675215 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee38e92-c346-4bd7-9555-b8ec8253f10a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ee38e92-c346-4bd7-9555-b8ec8253f10a" (UID: "9ee38e92-c346-4bd7-9555-b8ec8253f10a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:34:59.677008 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.676980 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee38e92-c346-4bd7-9555-b8ec8253f10a-kube-api-access-kldfv" (OuterVolumeSpecName: "kube-api-access-kldfv") pod "9ee38e92-c346-4bd7-9555-b8ec8253f10a" (UID: "9ee38e92-c346-4bd7-9555-b8ec8253f10a"). InnerVolumeSpecName "kube-api-access-kldfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:34:59.677186 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.677166 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf5f5318-9253-407d-adfc-ea741d4e26ca-kube-api-access-x9v4x" (OuterVolumeSpecName: "kube-api-access-x9v4x") pod "bf5f5318-9253-407d-adfc-ea741d4e26ca" (UID: "bf5f5318-9253-407d-adfc-ea741d4e26ca"). InnerVolumeSpecName "kube-api-access-x9v4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:34:59.776008 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.775914 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-openshift-service-ca\") pod \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " Sep 08 14:34:59.776183 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.776029 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-maas-api-tls\") pod \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " Sep 08 14:34:59.776183 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.776058 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n68mx\" (UniqueName: \"kubernetes.io/projected/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-kube-api-access-n68mx\") pod \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " Sep 08 14:34:59.776183 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.776126 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-metrics-certs\") pod \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\" (UID: \"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d\") " Sep 08 14:34:59.776399 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.776296 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d" (UID: "07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:34:59.776478 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.776462 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kldfv\" (UniqueName: \"kubernetes.io/projected/9ee38e92-c346-4bd7-9555-b8ec8253f10a-kube-api-access-kldfv\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:34:59.776539 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.776487 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:34:59.776539 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.776503 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf5f5318-9253-407d-adfc-ea741d4e26ca-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:34:59.776539 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.776519 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9v4x\" (UniqueName: \"kubernetes.io/projected/bf5f5318-9253-407d-adfc-ea741d4e26ca-kube-api-access-x9v4x\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:34:59.776539 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.776533 2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ee38e92-c346-4bd7-9555-b8ec8253f10a-plugins-config-volume\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:34:59.778696 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.778671 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-kube-api-access-n68mx" (OuterVolumeSpecName: "kube-api-access-n68mx") pod "07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d" (UID: "07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d"). InnerVolumeSpecName "kube-api-access-n68mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:34:59.778955 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.778930 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d" (UID: "07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:34:59.778955 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.778939 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d" (UID: "07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:34:59.877413 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.877378 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:34:59.877564 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.877428 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n68mx\" (UniqueName: \"kubernetes.io/projected/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-kube-api-access-n68mx\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:34:59.877564 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:34:59.877440 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:35:00.509077 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.508989 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" Sep 08 14:35:00.509077 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.508988 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h" event={"ID":"bf5f5318-9253-407d-adfc-ea741d4e26ca","Type":"ContainerDied","Data":"e2da609769ca96b5dec91b4a9d769f08f2b91fa3cf69d1900c528537ce0eb1ba"} Sep 08 14:35:00.509563 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.509136 2578 scope.go:117] "RemoveContainer" containerID="fbb5f8bb4729c203b0233d9635a7f7e80bda45e58b95b62e3f096ae2f3bce910" Sep 08 14:35:00.510475 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.510446 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" Sep 08 14:35:00.510554 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.510501 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2" event={"ID":"9ee38e92-c346-4bd7-9555-b8ec8253f10a","Type":"ContainerDied","Data":"86026de0da41a2c296f7d64d34ae7bf8c09570606d16fa7ad2e6266d02de8617"} Sep 08 14:35:00.512036 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.512014 2578 generic.go:356] "Generic (PLEG): container finished" podID="07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d" containerID="96e236a9ea01bf9291ccd02573f55e8d31bf35f0c0709e5b3bbde2799361b7bf" exitCode=0 Sep 08 14:35:00.512145 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.512049 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" Sep 08 14:35:00.512145 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.512074 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" event={"ID":"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d","Type":"ContainerDied","Data":"96e236a9ea01bf9291ccd02573f55e8d31bf35f0c0709e5b3bbde2799361b7bf"} Sep 08 14:35:00.512145 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.512107 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm" event={"ID":"07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d","Type":"ContainerDied","Data":"1d44d36c38675487565694dd02f7a83da6b72d17476f9b502d00ff3763b369c7"} Sep 08 14:35:00.519514 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.519496 2578 scope.go:117] "RemoveContainer" containerID="091fe0d1a62a946f4f189cad9ba769b3dc9de0b4ff702fcd98aeac72a6bd3b6e" Sep 08 14:35:00.530024 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.530002 2578 scope.go:117] "RemoveContainer" containerID="96e236a9ea01bf9291ccd02573f55e8d31bf35f0c0709e5b3bbde2799361b7bf" Sep 08 14:35:00.533219 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.533194 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h"] Sep 08 14:35:00.538616 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.538592 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-75b8cd-77d8c77c5-rw88h"] Sep 08 14:35:00.540138 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.540123 2578 scope.go:117] "RemoveContainer" containerID="96e236a9ea01bf9291ccd02573f55e8d31bf35f0c0709e5b3bbde2799361b7bf" Sep 08 14:35:00.540403 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:35:00.540384 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e236a9ea01bf9291ccd02573f55e8d31bf35f0c0709e5b3bbde2799361b7bf\": container with ID starting with 96e236a9ea01bf9291ccd02573f55e8d31bf35f0c0709e5b3bbde2799361b7bf not found: ID does not exist" containerID="96e236a9ea01bf9291ccd02573f55e8d31bf35f0c0709e5b3bbde2799361b7bf" Sep 08 14:35:00.540460 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.540414 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e236a9ea01bf9291ccd02573f55e8d31bf35f0c0709e5b3bbde2799361b7bf"} err="failed to get container status \"96e236a9ea01bf9291ccd02573f55e8d31bf35f0c0709e5b3bbde2799361b7bf\": rpc error: code = NotFound desc = could not find container \"96e236a9ea01bf9291ccd02573f55e8d31bf35f0c0709e5b3bbde2799361b7bf\": container with ID starting with 96e236a9ea01bf9291ccd02573f55e8d31bf35f0c0709e5b3bbde2799361b7bf not found: ID does not exist" Sep 08 14:35:00.547231 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.547210 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2"] Sep 08 14:35:00.551324 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.551304 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-75b8cd-77cf45897d-bd4c2"] Sep 08 14:35:00.561776 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.561758 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm"] Sep 08 14:35:00.568199 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:00.568181 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-75b8cd-798d65586f-xvpjm"] Sep 08 14:35:02.241756 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:02.241699 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d" path="/var/lib/kubelet/pods/07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d/volumes" Sep 08 14:35:02.242110 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:02.242086 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ee38e92-c346-4bd7-9555-b8ec8253f10a" path="/var/lib/kubelet/pods/9ee38e92-c346-4bd7-9555-b8ec8253f10a/volumes" Sep 08 14:35:02.242396 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:02.242383 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf5f5318-9253-407d-adfc-ea741d4e26ca" path="/var/lib/kubelet/pods/bf5f5318-9253-407d-adfc-ea741d4e26ca/volumes" Sep 08 14:35:05.953318 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:05.953282 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx"] Sep 08 14:35:05.953907 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:05.953533 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" podUID="6615426c-0495-4fc5-a631-75db3c59336a" containerName="maas-api" containerID="cri-o://42c8177678609ceed5fd1dcc58464692f58f177d84387e01307211b158bb2dae" gracePeriod=30 Sep 08 14:35:06.209709 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.209682 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:35:06.337790 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.337722 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6615426c-0495-4fc5-a631-75db3c59336a-openshift-service-ca\") pod \"6615426c-0495-4fc5-a631-75db3c59336a\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " Sep 08 14:35:06.337982 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.337802 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6615426c-0495-4fc5-a631-75db3c59336a-maas-api-tls\") pod \"6615426c-0495-4fc5-a631-75db3c59336a\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " Sep 08 14:35:06.337982 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.337898 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6615426c-0495-4fc5-a631-75db3c59336a-metrics-certs\") pod \"6615426c-0495-4fc5-a631-75db3c59336a\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " Sep 08 14:35:06.337982 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.337952 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krjd8\" (UniqueName: \"kubernetes.io/projected/6615426c-0495-4fc5-a631-75db3c59336a-kube-api-access-krjd8\") pod \"6615426c-0495-4fc5-a631-75db3c59336a\" (UID: \"6615426c-0495-4fc5-a631-75db3c59336a\") " Sep 08 14:35:06.338397 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.338253 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6615426c-0495-4fc5-a631-75db3c59336a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6615426c-0495-4fc5-a631-75db3c59336a" (UID: "6615426c-0495-4fc5-a631-75db3c59336a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:35:06.340961 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.340935 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6615426c-0495-4fc5-a631-75db3c59336a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6615426c-0495-4fc5-a631-75db3c59336a" (UID: "6615426c-0495-4fc5-a631-75db3c59336a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:35:06.341106 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.340955 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6615426c-0495-4fc5-a631-75db3c59336a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6615426c-0495-4fc5-a631-75db3c59336a" (UID: "6615426c-0495-4fc5-a631-75db3c59336a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:35:06.341106 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.341082 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6615426c-0495-4fc5-a631-75db3c59336a-kube-api-access-krjd8" (OuterVolumeSpecName: "kube-api-access-krjd8") pod "6615426c-0495-4fc5-a631-75db3c59336a" (UID: "6615426c-0495-4fc5-a631-75db3c59336a"). InnerVolumeSpecName "kube-api-access-krjd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:35:06.439616 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.439577 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6615426c-0495-4fc5-a631-75db3c59336a-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:35:06.439616 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.439608 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6615426c-0495-4fc5-a631-75db3c59336a-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:35:06.439616 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.439617 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6615426c-0495-4fc5-a631-75db3c59336a-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:35:06.439616 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.439625 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krjd8\" (UniqueName: \"kubernetes.io/projected/6615426c-0495-4fc5-a631-75db3c59336a-kube-api-access-krjd8\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:35:06.549689 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.549610 2578 generic.go:356] "Generic (PLEG): container finished" podID="6615426c-0495-4fc5-a631-75db3c59336a" containerID="42c8177678609ceed5fd1dcc58464692f58f177d84387e01307211b158bb2dae" exitCode=0 Sep 08 14:35:06.549689 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.549653 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" Sep 08 14:35:06.549894 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.549689 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" event={"ID":"6615426c-0495-4fc5-a631-75db3c59336a","Type":"ContainerDied","Data":"42c8177678609ceed5fd1dcc58464692f58f177d84387e01307211b158bb2dae"} Sep 08 14:35:06.549894 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.549759 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx" event={"ID":"6615426c-0495-4fc5-a631-75db3c59336a","Type":"ContainerDied","Data":"ec1356aec544e8c4466ebef0999ef8c29e855dd9b8fec5ba143d5c6d2eac76b3"} Sep 08 14:35:06.549894 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.549783 2578 scope.go:117] "RemoveContainer" containerID="42c8177678609ceed5fd1dcc58464692f58f177d84387e01307211b158bb2dae" Sep 08 14:35:06.559799 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.559784 2578 scope.go:117] "RemoveContainer" containerID="42c8177678609ceed5fd1dcc58464692f58f177d84387e01307211b158bb2dae" Sep 08 14:35:06.560065 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:35:06.560042 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c8177678609ceed5fd1dcc58464692f58f177d84387e01307211b158bb2dae\": container with ID starting with 42c8177678609ceed5fd1dcc58464692f58f177d84387e01307211b158bb2dae not found: ID does not exist" containerID="42c8177678609ceed5fd1dcc58464692f58f177d84387e01307211b158bb2dae" Sep 08 14:35:06.560169 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.560071 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c8177678609ceed5fd1dcc58464692f58f177d84387e01307211b158bb2dae"} err="failed to get container status \"42c8177678609ceed5fd1dcc58464692f58f177d84387e01307211b158bb2dae\": rpc error: code = NotFound desc = could not find container \"42c8177678609ceed5fd1dcc58464692f58f177d84387e01307211b158bb2dae\": container with ID starting with 42c8177678609ceed5fd1dcc58464692f58f177d84387e01307211b158bb2dae not found: ID does not exist" Sep 08 14:35:06.572404 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.572385 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx"] Sep 08 14:35:06.576356 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:06.576338 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8fb2c38-67894c7bf7-7lpjx"] Sep 08 14:35:08.242590 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:08.242555 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6615426c-0495-4fc5-a631-75db3c59336a" path="/var/lib/kubelet/pods/6615426c-0495-4fc5-a631-75db3c59336a/volumes" Sep 08 14:35:18.113067 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.113032 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl"] Sep 08 14:35:18.113667 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.113332 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" podUID="79a0c411-4969-4d42-811b-44a067de2c30" containerName="maas-api" containerID="cri-o://df72b65522604442a2351d536c8f70a6143e8b86e4acadc6d1a11e32d89c7579" gracePeriod=30 Sep 08 14:35:18.364070 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.364012 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:35:18.548619 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.548583 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9jn9\" (UniqueName: \"kubernetes.io/projected/79a0c411-4969-4d42-811b-44a067de2c30-kube-api-access-b9jn9\") pod \"79a0c411-4969-4d42-811b-44a067de2c30\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " Sep 08 14:35:18.548850 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.548709 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-metrics-certs\") pod \"79a0c411-4969-4d42-811b-44a067de2c30\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " Sep 08 14:35:18.548850 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.548815 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-maas-api-tls\") pod \"79a0c411-4969-4d42-811b-44a067de2c30\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " Sep 08 14:35:18.548972 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.548861 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a0c411-4969-4d42-811b-44a067de2c30-openshift-service-ca\") pod \"79a0c411-4969-4d42-811b-44a067de2c30\" (UID: \"79a0c411-4969-4d42-811b-44a067de2c30\") " Sep 08 14:35:18.549232 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.549205 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a0c411-4969-4d42-811b-44a067de2c30-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79a0c411-4969-4d42-811b-44a067de2c30" (UID: "79a0c411-4969-4d42-811b-44a067de2c30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:35:18.551020 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.550988 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "79a0c411-4969-4d42-811b-44a067de2c30" (UID: "79a0c411-4969-4d42-811b-44a067de2c30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:35:18.551110 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.551069 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a0c411-4969-4d42-811b-44a067de2c30-kube-api-access-b9jn9" (OuterVolumeSpecName: "kube-api-access-b9jn9") pod "79a0c411-4969-4d42-811b-44a067de2c30" (UID: "79a0c411-4969-4d42-811b-44a067de2c30"). InnerVolumeSpecName "kube-api-access-b9jn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:35:18.551110 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.551094 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79a0c411-4969-4d42-811b-44a067de2c30" (UID: "79a0c411-4969-4d42-811b-44a067de2c30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:35:18.619996 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.619928 2578 generic.go:356] "Generic (PLEG): container finished" podID="79a0c411-4969-4d42-811b-44a067de2c30" containerID="df72b65522604442a2351d536c8f70a6143e8b86e4acadc6d1a11e32d89c7579" exitCode=0 Sep 08 14:35:18.619996 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.619985 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" Sep 08 14:35:18.620142 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.620009 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" event={"ID":"79a0c411-4969-4d42-811b-44a067de2c30","Type":"ContainerDied","Data":"df72b65522604442a2351d536c8f70a6143e8b86e4acadc6d1a11e32d89c7579"} Sep 08 14:35:18.620142 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.620094 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl" event={"ID":"79a0c411-4969-4d42-811b-44a067de2c30","Type":"ContainerDied","Data":"76a8c8bdea4d6548b7f01b2d3870414a799b520837e8a6906de42d1005c133ce"} Sep 08 14:35:18.620142 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.620121 2578 scope.go:117] "RemoveContainer" containerID="df72b65522604442a2351d536c8f70a6143e8b86e4acadc6d1a11e32d89c7579" Sep 08 14:35:18.630361 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.630336 2578 scope.go:117] "RemoveContainer" containerID="df72b65522604442a2351d536c8f70a6143e8b86e4acadc6d1a11e32d89c7579" Sep 08 14:35:18.630751 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:35:18.630716 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df72b65522604442a2351d536c8f70a6143e8b86e4acadc6d1a11e32d89c7579\": container with ID starting with df72b65522604442a2351d536c8f70a6143e8b86e4acadc6d1a11e32d89c7579 not found: ID does not exist" containerID="df72b65522604442a2351d536c8f70a6143e8b86e4acadc6d1a11e32d89c7579" Sep 08 14:35:18.630838 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.630759 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df72b65522604442a2351d536c8f70a6143e8b86e4acadc6d1a11e32d89c7579"} err="failed to get container status \"df72b65522604442a2351d536c8f70a6143e8b86e4acadc6d1a11e32d89c7579\": rpc error: code = NotFound desc = could not find container \"df72b65522604442a2351d536c8f70a6143e8b86e4acadc6d1a11e32d89c7579\": container with ID starting with df72b65522604442a2351d536c8f70a6143e8b86e4acadc6d1a11e32d89c7579 not found: ID does not exist" Sep 08 14:35:18.644956 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.644935 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl"] Sep 08 14:35:18.649290 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.649271 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2a2de3-6fc98df4c4-b48sl"] Sep 08 14:35:18.649672 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.649656 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:35:18.649741 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.649677 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79a0c411-4969-4d42-811b-44a067de2c30-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:35:18.649741 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.649689 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9jn9\" (UniqueName: \"kubernetes.io/projected/79a0c411-4969-4d42-811b-44a067de2c30-kube-api-access-b9jn9\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:35:18.649741 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:18.649697 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79a0c411-4969-4d42-811b-44a067de2c30-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:35:20.241210 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:20.241175 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79a0c411-4969-4d42-811b-44a067de2c30" path="/var/lib/kubelet/pods/79a0c411-4969-4d42-811b-44a067de2c30/volumes" Sep 08 14:35:30.562204 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562164 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj"] Sep 08 14:35:30.562681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562567 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6615426c-0495-4fc5-a631-75db3c59336a" containerName="maas-api" Sep 08 14:35:30.562681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562579 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6615426c-0495-4fc5-a631-75db3c59336a" containerName="maas-api" Sep 08 14:35:30.562681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562591 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ee38e92-c346-4bd7-9555-b8ec8253f10a" containerName="payload-pre-processing" Sep 08 14:35:30.562681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562597 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ee38e92-c346-4bd7-9555-b8ec8253f10a" containerName="payload-pre-processing" Sep 08 14:35:30.562681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562608 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d" containerName="maas-api" Sep 08 14:35:30.562681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562614 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d" containerName="maas-api" Sep 08 14:35:30.562681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562619 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a0c411-4969-4d42-811b-44a067de2c30" containerName="maas-api" Sep 08 14:35:30.562681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562625 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a0c411-4969-4d42-811b-44a067de2c30" containerName="maas-api" Sep 08 14:35:30.562681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562637 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf5f5318-9253-407d-adfc-ea741d4e26ca" containerName="payload-processing" Sep 08 14:35:30.562681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562642 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf5f5318-9253-407d-adfc-ea741d4e26ca" containerName="payload-processing" Sep 08 14:35:30.563221 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562703 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ee38e92-c346-4bd7-9555-b8ec8253f10a" containerName="payload-pre-processing" Sep 08 14:35:30.563221 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562715 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="79a0c411-4969-4d42-811b-44a067de2c30" containerName="maas-api" Sep 08 14:35:30.563221 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562741 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="07d0a4c1-29ea-4a8d-8c2a-6470d37fe19d" containerName="maas-api" Sep 08 14:35:30.563221 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562750 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf5f5318-9253-407d-adfc-ea741d4e26ca" containerName="payload-processing" Sep 08 14:35:30.563221 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.562760 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6615426c-0495-4fc5-a631-75db3c59336a" containerName="maas-api" Sep 08 14:35:30.567778 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.567750 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:30.570254 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.570226 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-780944-serving-cert\"" Sep 08 14:35:30.575939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.575915 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj"] Sep 08 14:35:30.659883 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.659849 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:30.659883 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.659889 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t87zj\" (UniqueName: \"kubernetes.io/projected/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-kube-api-access-t87zj\") pod \"maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:30.660127 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.659951 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:30.660127 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.659979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:30.760452 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.760418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:30.760452 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.760456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t87zj\" (UniqueName: \"kubernetes.io/projected/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-kube-api-access-t87zj\") pod \"maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:30.760691 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.760493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:30.760691 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.760520 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:30.761209 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.761184 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:30.763040 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.763022 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:30.763099 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.763044 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:30.769901 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.769878 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t87zj\" (UniqueName: \"kubernetes.io/projected/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-kube-api-access-t87zj\") pod \"maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:30.880762 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:30.880638 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:31.154710 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:31.154668 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj"] Sep 08 14:35:31.156424 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:35:31.156387 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05126ef_a20b_4027_85a5_8f0ea79c7d8d.slice/crio-ab73b1a111b8f3819de8491cfb29db93d719fe62c9d24eed99c7bf4b3bcf0d65 WatchSource:0}: Error finding container ab73b1a111b8f3819de8491cfb29db93d719fe62c9d24eed99c7bf4b3bcf0d65: Status 404 returned error can't find the container with id ab73b1a111b8f3819de8491cfb29db93d719fe62c9d24eed99c7bf4b3bcf0d65 Sep 08 14:35:31.693531 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:31.693489 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" event={"ID":"b05126ef-a20b-4027-85a5-8f0ea79c7d8d","Type":"ContainerStarted","Data":"ab73b1a111b8f3819de8491cfb29db93d719fe62c9d24eed99c7bf4b3bcf0d65"} Sep 08 14:35:32.701771 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:32.701714 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" event={"ID":"b05126ef-a20b-4027-85a5-8f0ea79c7d8d","Type":"ContainerStarted","Data":"38e801b5e116e50b6d29efba1255eb86939be8cf04988a9060c1464740c6a185"} Sep 08 14:35:32.702234 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:32.701854 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:32.729109 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:32.729059 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" podStartSLOduration=2.125390768 podStartE2EDuration="2.729045491s" totalImagesPullingTime="603.654723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:35:32.727471959 +0000 UTC m=+1145.011789074" watchObservedRunningTime="2026-09-08 14:35:32.729045491 +0000 UTC m=+1145.013362605" Sep 08 14:35:33.364504 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.364457 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm"] Sep 08 14:35:33.368349 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.368330 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:33.370513 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.370487 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-23a1d4-serving-cert\"" Sep 08 14:35:33.378506 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.378480 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm"] Sep 08 14:35:33.485649 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.485616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:33.485936 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.485708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:33.485936 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.485747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbhs6\" (UniqueName: \"kubernetes.io/projected/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-kube-api-access-cbhs6\") pod \"maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:33.485936 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.485809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:33.586293 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.586251 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:33.586469 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.586313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:33.586469 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.586372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:33.586469 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.586391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbhs6\" (UniqueName: \"kubernetes.io/projected/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-kube-api-access-cbhs6\") pod \"maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:33.586469 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:35:33.586410 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-serving-cert: secret "maas-api-e2e-shared-a-w2-23a1d4-serving-cert" not found Sep 08 14:35:33.586709 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:35:33.586484 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-maas-api-tls podName:393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285 nodeName:}" failed. No retries permitted until 2026-09-08 14:35:34.08646334 +0000 UTC m=+1146.370780451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-maas-api-tls") pod "maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" (UID: "393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285") : secret "maas-api-e2e-shared-a-w2-23a1d4-serving-cert" not found Sep 08 14:35:33.587018 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.586999 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:33.589161 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.589143 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:33.594623 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:33.594604 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbhs6\" (UniqueName: \"kubernetes.io/projected/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-kube-api-access-cbhs6\") pod \"maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:34.091146 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:34.091071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:34.094225 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:34.094198 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:34.280990 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:34.280960 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:34.558934 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:34.558911 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm"] Sep 08 14:35:34.560331 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:35:34.560307 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393fc1d9_bbe5_4c7b_a63b_8e1b3ccfe285.slice/crio-be70896b14ce848a065d594971c55e30e286a634e2000f774c8b79b639193db0 WatchSource:0}: Error finding container be70896b14ce848a065d594971c55e30e286a634e2000f774c8b79b639193db0: Status 404 returned error can't find the container with id be70896b14ce848a065d594971c55e30e286a634e2000f774c8b79b639193db0 Sep 08 14:35:34.715551 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:34.715519 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" event={"ID":"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285","Type":"ContainerStarted","Data":"be70896b14ce848a065d594971c55e30e286a634e2000f774c8b79b639193db0"} Sep 08 14:35:35.723180 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:35.723147 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" event={"ID":"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285","Type":"ContainerStarted","Data":"7427d90d17bdf35f7cb252bbbbd1ef6390b41fcf6ae95205e8507dae86e2e6a8"} Sep 08 14:35:35.723638 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:35.723211 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:35.741814 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:35.741767 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" podStartSLOduration=2.401818332 podStartE2EDuration="2.741751759s" totalImagesPullingTime="339.933427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:35:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:35:35.739414488 +0000 UTC m=+1148.023731602" watchObservedRunningTime="2026-09-08 14:35:35.741751759 +0000 UTC m=+1148.026068875" Sep 08 14:35:38.713875 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:38.713845 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:35:41.734771 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:41.734713 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:35:58.895239 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:58.895164 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw"] Sep 08 14:35:58.899106 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:58.899088 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:58.901450 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:58.901428 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-017dbd-serving-cert\"" Sep 08 14:35:58.908473 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:58.908433 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw"] Sep 08 14:35:59.017411 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.017373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5d4802-46e4-463a-8db9-01bd3f35768c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:59.017594 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.017503 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5d4802-46e4-463a-8db9-01bd3f35768c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:59.017594 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.017545 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5d4802-46e4-463a-8db9-01bd3f35768c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:59.017594 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.017571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9ddm\" (UniqueName: \"kubernetes.io/projected/4a5d4802-46e4-463a-8db9-01bd3f35768c-kube-api-access-w9ddm\") pod \"maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:59.118326 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.118302 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5d4802-46e4-463a-8db9-01bd3f35768c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:59.118497 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.118337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5d4802-46e4-463a-8db9-01bd3f35768c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:59.118497 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.118361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9ddm\" (UniqueName: \"kubernetes.io/projected/4a5d4802-46e4-463a-8db9-01bd3f35768c-kube-api-access-w9ddm\") pod \"maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:59.118497 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.118428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5d4802-46e4-463a-8db9-01bd3f35768c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:59.119381 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.119354 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5d4802-46e4-463a-8db9-01bd3f35768c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:59.120912 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.120880 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5d4802-46e4-463a-8db9-01bd3f35768c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:59.121058 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.121037 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5d4802-46e4-463a-8db9-01bd3f35768c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:59.128859 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.128835 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9ddm\" (UniqueName: \"kubernetes.io/projected/4a5d4802-46e4-463a-8db9-01bd3f35768c-kube-api-access-w9ddm\") pod \"maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:59.211715 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.211687 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:35:59.484961 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.484930 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw"] Sep 08 14:35:59.485910 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:35:59.485871 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5d4802_46e4_463a_8db9_01bd3f35768c.slice/crio-d252fa19cdb1a28d3eebbb3c5f81170b76a4ff104c66c20d1ab7cdc09e9e34dd WatchSource:0}: Error finding container d252fa19cdb1a28d3eebbb3c5f81170b76a4ff104c66c20d1ab7cdc09e9e34dd: Status 404 returned error can't find the container with id d252fa19cdb1a28d3eebbb3c5f81170b76a4ff104c66c20d1ab7cdc09e9e34dd Sep 08 14:35:59.868342 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:35:59.868253 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" event={"ID":"4a5d4802-46e4-463a-8db9-01bd3f35768c","Type":"ContainerStarted","Data":"d252fa19cdb1a28d3eebbb3c5f81170b76a4ff104c66c20d1ab7cdc09e9e34dd"} Sep 08 14:36:00.875279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:00.875191 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" event={"ID":"4a5d4802-46e4-463a-8db9-01bd3f35768c","Type":"ContainerStarted","Data":"afeeb00d6b9439b71a9ef2b093d28a2e67a7e98ddd040dbb58c55c5906b0cdab"} Sep 08 14:36:00.875279 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:00.875250 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:36:00.893067 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:00.893019 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" podStartSLOduration=1.765180044 podStartE2EDuration="2.893004095s" totalImagesPullingTime="1.127824051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:35:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:36:00.89070547 +0000 UTC m=+1173.175022586" watchObservedRunningTime="2026-09-08 14:36:00.893004095 +0000 UTC m=+1173.177321210" Sep 08 14:36:01.748189 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.748157 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98"] Sep 08 14:36:01.751977 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.751951 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:01.754070 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.754048 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-34656b-serving-cert\"" Sep 08 14:36:01.761065 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.761012 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98"] Sep 08 14:36:01.846228 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.846191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlmtw\" (UniqueName: \"kubernetes.io/projected/ff62c013-0f6e-461e-9f8a-90faed677687-kube-api-access-hlmtw\") pod \"maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:01.846228 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.846231 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:01.846492 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.846348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:01.846492 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.846392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff62c013-0f6e-461e-9f8a-90faed677687-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:01.948000 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.947962 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlmtw\" (UniqueName: \"kubernetes.io/projected/ff62c013-0f6e-461e-9f8a-90faed677687-kube-api-access-hlmtw\") pod \"maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:01.948449 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.948008 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:01.948449 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.948082 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:01.948449 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.948135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff62c013-0f6e-461e-9f8a-90faed677687-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:01.948449 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:01.948231 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-serving-cert: secret "maas-api-e2e-shared-b-w2-34656b-serving-cert" not found Sep 08 14:36:01.948449 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:01.948314 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-maas-api-tls podName:ff62c013-0f6e-461e-9f8a-90faed677687 nodeName:}" failed. No retries permitted until 2026-09-08 14:36:02.448290329 +0000 UTC m=+1174.732607442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-maas-api-tls") pod "maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" (UID: "ff62c013-0f6e-461e-9f8a-90faed677687") : secret "maas-api-e2e-shared-b-w2-34656b-serving-cert" not found Sep 08 14:36:01.948842 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.948824 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff62c013-0f6e-461e-9f8a-90faed677687-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:01.950634 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.950617 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:01.958375 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:01.958343 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlmtw\" (UniqueName: \"kubernetes.io/projected/ff62c013-0f6e-461e-9f8a-90faed677687-kube-api-access-hlmtw\") pod \"maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:02.453492 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:02.453456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:02.456132 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:02.456106 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:02.665469 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:02.665432 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:02.945873 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:02.945456 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98"] Sep 08 14:36:03.893645 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:03.893609 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" event={"ID":"ff62c013-0f6e-461e-9f8a-90faed677687","Type":"ContainerStarted","Data":"131c040b7386c8676f798b0fa411890aa23e78f3ada17c78415c44b279bc8c2d"} Sep 08 14:36:03.894137 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:03.893667 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" event={"ID":"ff62c013-0f6e-461e-9f8a-90faed677687","Type":"ContainerStarted","Data":"9844cc327ce9f161e21d2d2017b8c11afe966c830e8a276621b0fca631403db1"} Sep 08 14:36:03.894137 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:03.893772 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:03.912861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:03.912810 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" podStartSLOduration=2.49994695 podStartE2EDuration="2.912796701s" totalImagesPullingTime="412.849751ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:36:03.910526977 +0000 UTC m=+1176.194844107" watchObservedRunningTime="2026-09-08 14:36:03.912796701 +0000 UTC m=+1176.197113816" Sep 08 14:36:06.887113 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:06.887081 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:36:09.904982 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:09.904949 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:36:26.333759 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:26.333711 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7895-9d261d"] Sep 08 14:36:26.337486 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:26.337463 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-9d261d" Sep 08 14:36:26.340282 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:26.340259 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qcp6n\"" Sep 08 14:36:26.343032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:26.342999 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7895-9d261d"] Sep 08 14:36:26.355482 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:26.355452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88kw\" (UniqueName: \"kubernetes.io/projected/02cc6144-ab18-4866-bba2-d34ec003763c-kube-api-access-b88kw\") pod \"test-curl-7895-9d261d\" (UID: \"02cc6144-ab18-4866-bba2-d34ec003763c\") " pod="opendatahub/test-curl-7895-9d261d" Sep 08 14:36:26.456688 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:26.456644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88kw\" (UniqueName: \"kubernetes.io/projected/02cc6144-ab18-4866-bba2-d34ec003763c-kube-api-access-b88kw\") pod \"test-curl-7895-9d261d\" (UID: \"02cc6144-ab18-4866-bba2-d34ec003763c\") " pod="opendatahub/test-curl-7895-9d261d" Sep 08 14:36:26.465133 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:26.465106 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88kw\" (UniqueName: \"kubernetes.io/projected/02cc6144-ab18-4866-bba2-d34ec003763c-kube-api-access-b88kw\") pod \"test-curl-7895-9d261d\" (UID: \"02cc6144-ab18-4866-bba2-d34ec003763c\") " pod="opendatahub/test-curl-7895-9d261d" Sep 08 14:36:26.647953 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:26.647855 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-9d261d" Sep 08 14:36:26.921104 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:26.921081 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7895-9d261d"] Sep 08 14:36:26.922622 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:36:26.922599 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02cc6144_ab18_4866_bba2_d34ec003763c.slice/crio-e44f72027298412769f1f5f7a870cfd6f6419725314debbd924dfa82ca6caed5 WatchSource:0}: Error finding container e44f72027298412769f1f5f7a870cfd6f6419725314debbd924dfa82ca6caed5: Status 404 returned error can't find the container with id e44f72027298412769f1f5f7a870cfd6f6419725314debbd924dfa82ca6caed5 Sep 08 14:36:27.025039 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.025006 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7895-9d261d" event={"ID":"02cc6144-ab18-4866-bba2-d34ec003763c","Type":"ContainerStarted","Data":"12bf4a7bf37a203da05bfe816b4ee39ae3a6d328321fd8c084faf7ede3085e7b"} Sep 08 14:36:27.025039 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.025046 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7895-9d261d" event={"ID":"02cc6144-ab18-4866-bba2-d34ec003763c","Type":"ContainerStarted","Data":"e44f72027298412769f1f5f7a870cfd6f6419725314debbd924dfa82ca6caed5"} Sep 08 14:36:27.039269 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.039224 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7895-9d261d" podStartSLOduration=1.03921022 podStartE2EDuration="1.03921022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:36:27.038113073 +0000 UTC m=+1199.322430187" watchObservedRunningTime="2026-09-08 14:36:27.03921022 +0000 UTC m=+1199.323527373" Sep 08 14:36:27.641876 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.641833 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7895-9d261d"] Sep 08 14:36:27.647523 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.647494 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7895-9d261d"] Sep 08 14:36:27.773752 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.773689 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7895-28dd92"] Sep 08 14:36:27.774197 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.774178 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02cc6144-ab18-4866-bba2-d34ec003763c" containerName="test-curl-7895-9d261d" Sep 08 14:36:27.774290 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.774200 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="02cc6144-ab18-4866-bba2-d34ec003763c" containerName="test-curl-7895-9d261d" Sep 08 14:36:27.774353 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.774338 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="02cc6144-ab18-4866-bba2-d34ec003763c" containerName="test-curl-7895-9d261d" Sep 08 14:36:27.777627 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.777593 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-28dd92" Sep 08 14:36:27.781702 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.781676 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7895-28dd92"] Sep 08 14:36:27.870559 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.870524 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2r2c\" (UniqueName: \"kubernetes.io/projected/76f62ac1-e672-4fb7-9a64-fba3148f73f3-kube-api-access-s2r2c\") pod \"test-curl-7895-28dd92\" (UID: \"76f62ac1-e672-4fb7-9a64-fba3148f73f3\") " pod="opendatahub/test-curl-7895-28dd92" Sep 08 14:36:27.971452 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.971422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2r2c\" (UniqueName: \"kubernetes.io/projected/76f62ac1-e672-4fb7-9a64-fba3148f73f3-kube-api-access-s2r2c\") pod \"test-curl-7895-28dd92\" (UID: \"76f62ac1-e672-4fb7-9a64-fba3148f73f3\") " pod="opendatahub/test-curl-7895-28dd92" Sep 08 14:36:27.979957 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:27.979932 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2r2c\" (UniqueName: \"kubernetes.io/projected/76f62ac1-e672-4fb7-9a64-fba3148f73f3-kube-api-access-s2r2c\") pod \"test-curl-7895-28dd92\" (UID: \"76f62ac1-e672-4fb7-9a64-fba3148f73f3\") " pod="opendatahub/test-curl-7895-28dd92" Sep 08 14:36:28.030660 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:28.030635 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7895-9d261d" container="test-curl-7895-9d261d" Sep 08 14:36:28.087593 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:28.087567 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-28dd92" Sep 08 14:36:28.250208 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:28.250132 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:36:28.251955 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:28.251933 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:36:28.257825 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:28.257799 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:36:28.259799 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:28.259783 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:36:28.362424 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:28.362395 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7895-28dd92"] Sep 08 14:36:28.364705 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:36:28.364682 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76f62ac1_e672_4fb7_9a64_fba3148f73f3.slice/crio-6440b8800cbac2947ac144693162552661340b9748f9a5f7be95cf58412a1003 WatchSource:0}: Error finding container 6440b8800cbac2947ac144693162552661340b9748f9a5f7be95cf58412a1003: Status 404 returned error can't find the container with id 6440b8800cbac2947ac144693162552661340b9748f9a5f7be95cf58412a1003 Sep 08 14:36:29.037316 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.037274 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7895-28dd92" event={"ID":"76f62ac1-e672-4fb7-9a64-fba3148f73f3","Type":"ContainerStarted","Data":"16d483178632129841b24ad5e3fb04dd59e9a73a1fdf61922c53d5b763e65792"} Sep 08 14:36:29.037316 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.037323 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7895-28dd92" event={"ID":"76f62ac1-e672-4fb7-9a64-fba3148f73f3","Type":"ContainerStarted","Data":"6440b8800cbac2947ac144693162552661340b9748f9a5f7be95cf58412a1003"} Sep 08 14:36:29.037827 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.037344 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7895-9d261d" podUID="02cc6144-ab18-4866-bba2-d34ec003763c" containerName="test-curl-7895-9d261d" containerID="cri-o://12bf4a7bf37a203da05bfe816b4ee39ae3a6d328321fd8c084faf7ede3085e7b" gracePeriod=2 Sep 08 14:36:29.052135 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.052088 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7895-28dd92" podStartSLOduration=2.052073524 podStartE2EDuration="2.052073524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:36:29.051009925 +0000 UTC m=+1201.335327040" watchObservedRunningTime="2026-09-08 14:36:29.052073524 +0000 UTC m=+1201.336390668" Sep 08 14:36:29.659778 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.659745 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7895-28dd92"] Sep 08 14:36:29.665310 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.665281 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7895-28dd92"] Sep 08 14:36:29.784617 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.784586 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7895-37fa23"] Sep 08 14:36:29.785020 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.785006 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76f62ac1-e672-4fb7-9a64-fba3148f73f3" containerName="test-curl-7895-28dd92" Sep 08 14:36:29.785073 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.785021 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="76f62ac1-e672-4fb7-9a64-fba3148f73f3" containerName="test-curl-7895-28dd92" Sep 08 14:36:29.785115 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.785106 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="76f62ac1-e672-4fb7-9a64-fba3148f73f3" containerName="test-curl-7895-28dd92" Sep 08 14:36:29.788142 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.788126 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-37fa23" Sep 08 14:36:29.794043 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.794016 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7895-37fa23"] Sep 08 14:36:29.890052 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.890023 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms7bn\" (UniqueName: \"kubernetes.io/projected/cd77d94e-3f5a-4aac-ac5b-1b26a195e78a-kube-api-access-ms7bn\") pod \"test-curl-7895-37fa23\" (UID: \"cd77d94e-3f5a-4aac-ac5b-1b26a195e78a\") " pod="opendatahub/test-curl-7895-37fa23" Sep 08 14:36:29.991205 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.991176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms7bn\" (UniqueName: \"kubernetes.io/projected/cd77d94e-3f5a-4aac-ac5b-1b26a195e78a-kube-api-access-ms7bn\") pod \"test-curl-7895-37fa23\" (UID: \"cd77d94e-3f5a-4aac-ac5b-1b26a195e78a\") " pod="opendatahub/test-curl-7895-37fa23" Sep 08 14:36:29.999742 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:29.999708 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms7bn\" (UniqueName: \"kubernetes.io/projected/cd77d94e-3f5a-4aac-ac5b-1b26a195e78a-kube-api-access-ms7bn\") pod \"test-curl-7895-37fa23\" (UID: \"cd77d94e-3f5a-4aac-ac5b-1b26a195e78a\") " pod="opendatahub/test-curl-7895-37fa23" Sep 08 14:36:30.042815 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:30.042758 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7895-28dd92" container="test-curl-7895-28dd92" Sep 08 14:36:30.099062 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:30.099040 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-37fa23" Sep 08 14:36:30.369626 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:30.369601 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7895-37fa23"] Sep 08 14:36:30.370698 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:36:30.370673 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd77d94e_3f5a_4aac_ac5b_1b26a195e78a.slice/crio-85db9f29f2da3d8c34fd7270bfa086e16f741249c887aaaf6d9e8131f217ffa5 WatchSource:0}: Error finding container 85db9f29f2da3d8c34fd7270bfa086e16f741249c887aaaf6d9e8131f217ffa5: Status 404 returned error can't find the container with id 85db9f29f2da3d8c34fd7270bfa086e16f741249c887aaaf6d9e8131f217ffa5 Sep 08 14:36:31.051187 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.051148 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7895-37fa23" event={"ID":"cd77d94e-3f5a-4aac-ac5b-1b26a195e78a","Type":"ContainerStarted","Data":"c43d47817bd2b5c80b536c42c42bea5a9506d8a9bb23d0b2278724625d033f89"} Sep 08 14:36:31.051596 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.051210 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7895-37fa23" event={"ID":"cd77d94e-3f5a-4aac-ac5b-1b26a195e78a","Type":"ContainerStarted","Data":"85db9f29f2da3d8c34fd7270bfa086e16f741249c887aaaf6d9e8131f217ffa5"} Sep 08 14:36:31.051596 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.051246 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7895-28dd92" podUID="76f62ac1-e672-4fb7-9a64-fba3148f73f3" containerName="test-curl-7895-28dd92" containerID="cri-o://16d483178632129841b24ad5e3fb04dd59e9a73a1fdf61922c53d5b763e65792" gracePeriod=2 Sep 08 14:36:31.065171 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.065126 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7895-37fa23" podStartSLOduration=2.065111388 podStartE2EDuration="2.065111388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:36:31.063562957 +0000 UTC m=+1203.347880070" watchObservedRunningTime="2026-09-08 14:36:31.065111388 +0000 UTC m=+1203.349428502" Sep 08 14:36:31.211194 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.211168 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-9d261d" Sep 08 14:36:31.303318 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.303250 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b88kw\" (UniqueName: \"kubernetes.io/projected/02cc6144-ab18-4866-bba2-d34ec003763c-kube-api-access-b88kw\") pod \"02cc6144-ab18-4866-bba2-d34ec003763c\" (UID: \"02cc6144-ab18-4866-bba2-d34ec003763c\") " Sep 08 14:36:31.305253 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.305226 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02cc6144-ab18-4866-bba2-d34ec003763c-kube-api-access-b88kw" (OuterVolumeSpecName: "kube-api-access-b88kw") pod "02cc6144-ab18-4866-bba2-d34ec003763c" (UID: "02cc6144-ab18-4866-bba2-d34ec003763c"). InnerVolumeSpecName "kube-api-access-b88kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:36:31.405061 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.405024 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b88kw\" (UniqueName: \"kubernetes.io/projected/02cc6144-ab18-4866-bba2-d34ec003763c-kube-api-access-b88kw\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:36:31.670246 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.670161 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7895-37fa23"] Sep 08 14:36:31.675335 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.675305 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7895-37fa23"] Sep 08 14:36:31.795603 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.795573 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7895-fdabeb"] Sep 08 14:36:31.796029 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.796015 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd77d94e-3f5a-4aac-ac5b-1b26a195e78a" containerName="test-curl-7895-37fa23" Sep 08 14:36:31.796076 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.796031 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd77d94e-3f5a-4aac-ac5b-1b26a195e78a" containerName="test-curl-7895-37fa23" Sep 08 14:36:31.796117 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.796099 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd77d94e-3f5a-4aac-ac5b-1b26a195e78a" containerName="test-curl-7895-37fa23" Sep 08 14:36:31.799194 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.799178 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-fdabeb" Sep 08 14:36:31.804984 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.804959 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7895-fdabeb"] Sep 08 14:36:31.908549 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:31.908515 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnn69\" (UniqueName: \"kubernetes.io/projected/97c26fe5-d1e3-419d-9e76-7b529e33349e-kube-api-access-qnn69\") pod \"test-curl-7895-fdabeb\" (UID: \"97c26fe5-d1e3-419d-9e76-7b529e33349e\") " pod="opendatahub/test-curl-7895-fdabeb" Sep 08 14:36:32.009454 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:32.009419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnn69\" (UniqueName: \"kubernetes.io/projected/97c26fe5-d1e3-419d-9e76-7b529e33349e-kube-api-access-qnn69\") pod \"test-curl-7895-fdabeb\" (UID: \"97c26fe5-d1e3-419d-9e76-7b529e33349e\") " pod="opendatahub/test-curl-7895-fdabeb" Sep 08 14:36:32.018945 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:32.018917 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnn69\" (UniqueName: \"kubernetes.io/projected/97c26fe5-d1e3-419d-9e76-7b529e33349e-kube-api-access-qnn69\") pod \"test-curl-7895-fdabeb\" (UID: \"97c26fe5-d1e3-419d-9e76-7b529e33349e\") " pod="opendatahub/test-curl-7895-fdabeb" Sep 08 14:36:32.059346 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:32.059316 2578 generic.go:356] "Generic (PLEG): container finished" podID="02cc6144-ab18-4866-bba2-d34ec003763c" containerID="12bf4a7bf37a203da05bfe816b4ee39ae3a6d328321fd8c084faf7ede3085e7b" exitCode=137 Sep 08 14:36:32.059781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:32.059429 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-9d261d" Sep 08 14:36:32.059781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:32.059483 2578 scope.go:117] "RemoveContainer" containerID="12bf4a7bf37a203da05bfe816b4ee39ae3a6d328321fd8c084faf7ede3085e7b" Sep 08 14:36:32.059897 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:32.059802 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7895-37fa23" container="test-curl-7895-37fa23" Sep 08 14:36:32.069638 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:32.069623 2578 scope.go:117] "RemoveContainer" containerID="12bf4a7bf37a203da05bfe816b4ee39ae3a6d328321fd8c084faf7ede3085e7b" Sep 08 14:36:32.069947 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:32.069926 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12bf4a7bf37a203da05bfe816b4ee39ae3a6d328321fd8c084faf7ede3085e7b\": container with ID starting with 12bf4a7bf37a203da05bfe816b4ee39ae3a6d328321fd8c084faf7ede3085e7b not found: ID does not exist" containerID="12bf4a7bf37a203da05bfe816b4ee39ae3a6d328321fd8c084faf7ede3085e7b" Sep 08 14:36:32.070023 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:32.069960 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12bf4a7bf37a203da05bfe816b4ee39ae3a6d328321fd8c084faf7ede3085e7b"} err="failed to get container status \"12bf4a7bf37a203da05bfe816b4ee39ae3a6d328321fd8c084faf7ede3085e7b\": rpc error: code = NotFound desc = could not find container \"12bf4a7bf37a203da05bfe816b4ee39ae3a6d328321fd8c084faf7ede3085e7b\": container with ID starting with 12bf4a7bf37a203da05bfe816b4ee39ae3a6d328321fd8c084faf7ede3085e7b not found: ID does not exist" Sep 08 14:36:32.070228 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:32.070209 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7895-9d261d\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="02cc6144-ab18-4866-bba2-d34ec003763c" pod="opendatahub/test-curl-7895-9d261d" Sep 08 14:36:32.111657 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:32.111634 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-fdabeb" Sep 08 14:36:32.242236 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:32.242204 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02cc6144-ab18-4866-bba2-d34ec003763c" path="/var/lib/kubelet/pods/02cc6144-ab18-4866-bba2-d34ec003763c/volumes" Sep 08 14:36:32.391002 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:32.390973 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7895-fdabeb"] Sep 08 14:36:32.392994 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:36:32.392966 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97c26fe5_d1e3_419d_9e76_7b529e33349e.slice/crio-6df03881d8ece21e71308a4c0b3167d3b03a4c2cbec600734814683d97122a8b WatchSource:0}: Error finding container 6df03881d8ece21e71308a4c0b3167d3b03a4c2cbec600734814683d97122a8b: Status 404 returned error can't find the container with id 6df03881d8ece21e71308a4c0b3167d3b03a4c2cbec600734814683d97122a8b Sep 08 14:36:33.067586 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:33.067553 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7895-fdabeb" event={"ID":"97c26fe5-d1e3-419d-9e76-7b529e33349e","Type":"ContainerStarted","Data":"2e790461a0eba8ba9698fa37b30da7ef774b3726374a914c3229d52fcd2864b8"} Sep 08 14:36:33.067939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:33.067604 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7895-fdabeb" event={"ID":"97c26fe5-d1e3-419d-9e76-7b529e33349e","Type":"ContainerStarted","Data":"6df03881d8ece21e71308a4c0b3167d3b03a4c2cbec600734814683d97122a8b"} Sep 08 14:36:33.068832 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:33.068805 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7895-37fa23" podUID="cd77d94e-3f5a-4aac-ac5b-1b26a195e78a" containerName="test-curl-7895-37fa23" containerID="cri-o://c43d47817bd2b5c80b536c42c42bea5a9506d8a9bb23d0b2278724625d033f89" gracePeriod=2 Sep 08 14:36:33.083637 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:33.083591 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7895-fdabeb" podStartSLOduration=2.083576228 podStartE2EDuration="2.083576228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:36:33.081849905 +0000 UTC m=+1205.366167024" watchObservedRunningTime="2026-09-08 14:36:33.083576228 +0000 UTC m=+1205.367893343" Sep 08 14:36:33.199927 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:33.199904 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-28dd92" Sep 08 14:36:33.322708 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:33.322644 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2r2c\" (UniqueName: \"kubernetes.io/projected/76f62ac1-e672-4fb7-9a64-fba3148f73f3-kube-api-access-s2r2c\") pod \"76f62ac1-e672-4fb7-9a64-fba3148f73f3\" (UID: \"76f62ac1-e672-4fb7-9a64-fba3148f73f3\") " Sep 08 14:36:33.324768 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:33.324720 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f62ac1-e672-4fb7-9a64-fba3148f73f3-kube-api-access-s2r2c" (OuterVolumeSpecName: "kube-api-access-s2r2c") pod "76f62ac1-e672-4fb7-9a64-fba3148f73f3" (UID: "76f62ac1-e672-4fb7-9a64-fba3148f73f3"). InnerVolumeSpecName "kube-api-access-s2r2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:36:33.423585 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:33.423558 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2r2c\" (UniqueName: \"kubernetes.io/projected/76f62ac1-e672-4fb7-9a64-fba3148f73f3-kube-api-access-s2r2c\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:36:33.701443 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:33.701358 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7895-fdabeb"] Sep 08 14:36:33.706074 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:33.706044 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7895-fdabeb"] Sep 08 14:36:34.075820 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:34.075778 2578 generic.go:356] "Generic (PLEG): container finished" podID="76f62ac1-e672-4fb7-9a64-fba3148f73f3" containerID="16d483178632129841b24ad5e3fb04dd59e9a73a1fdf61922c53d5b763e65792" exitCode=137 Sep 08 14:36:34.076278 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:34.075842 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-28dd92" Sep 08 14:36:34.076278 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:34.075929 2578 scope.go:117] "RemoveContainer" containerID="16d483178632129841b24ad5e3fb04dd59e9a73a1fdf61922c53d5b763e65792" Sep 08 14:36:34.076278 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:34.076142 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7895-fdabeb" container="test-curl-7895-fdabeb" Sep 08 14:36:34.085238 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:34.085223 2578 scope.go:117] "RemoveContainer" containerID="16d483178632129841b24ad5e3fb04dd59e9a73a1fdf61922c53d5b763e65792" Sep 08 14:36:34.085494 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:34.085472 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d483178632129841b24ad5e3fb04dd59e9a73a1fdf61922c53d5b763e65792\": container with ID starting with 16d483178632129841b24ad5e3fb04dd59e9a73a1fdf61922c53d5b763e65792 not found: ID does not exist" containerID="16d483178632129841b24ad5e3fb04dd59e9a73a1fdf61922c53d5b763e65792" Sep 08 14:36:34.085569 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:34.085501 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d483178632129841b24ad5e3fb04dd59e9a73a1fdf61922c53d5b763e65792"} err="failed to get container status \"16d483178632129841b24ad5e3fb04dd59e9a73a1fdf61922c53d5b763e65792\": rpc error: code = NotFound desc = could not find container \"16d483178632129841b24ad5e3fb04dd59e9a73a1fdf61922c53d5b763e65792\": container with ID starting with 16d483178632129841b24ad5e3fb04dd59e9a73a1fdf61922c53d5b763e65792 not found: ID does not exist" Sep 08 14:36:34.085798 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:34.085773 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7895-28dd92\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="76f62ac1-e672-4fb7-9a64-fba3148f73f3" pod="opendatahub/test-curl-7895-28dd92" Sep 08 14:36:34.242268 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:34.242238 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76f62ac1-e672-4fb7-9a64-fba3148f73f3" path="/var/lib/kubelet/pods/76f62ac1-e672-4fb7-9a64-fba3148f73f3/volumes" Sep 08 14:36:35.083848 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:35.083817 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7895-fdabeb" podUID="97c26fe5-d1e3-419d-9e76-7b529e33349e" containerName="test-curl-7895-fdabeb" containerID="cri-o://2e790461a0eba8ba9698fa37b30da7ef774b3726374a914c3229d52fcd2864b8" gracePeriod=2 Sep 08 14:36:35.250072 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:35.250048 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-37fa23" Sep 08 14:36:35.342596 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:35.342509 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms7bn\" (UniqueName: \"kubernetes.io/projected/cd77d94e-3f5a-4aac-ac5b-1b26a195e78a-kube-api-access-ms7bn\") pod \"cd77d94e-3f5a-4aac-ac5b-1b26a195e78a\" (UID: \"cd77d94e-3f5a-4aac-ac5b-1b26a195e78a\") " Sep 08 14:36:35.344800 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:35.344772 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd77d94e-3f5a-4aac-ac5b-1b26a195e78a-kube-api-access-ms7bn" (OuterVolumeSpecName: "kube-api-access-ms7bn") pod "cd77d94e-3f5a-4aac-ac5b-1b26a195e78a" (UID: "cd77d94e-3f5a-4aac-ac5b-1b26a195e78a"). InnerVolumeSpecName "kube-api-access-ms7bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:36:35.443341 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:35.443308 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms7bn\" (UniqueName: \"kubernetes.io/projected/cd77d94e-3f5a-4aac-ac5b-1b26a195e78a-kube-api-access-ms7bn\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:36:36.091222 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:36.091184 2578 generic.go:356] "Generic (PLEG): container finished" podID="cd77d94e-3f5a-4aac-ac5b-1b26a195e78a" containerID="c43d47817bd2b5c80b536c42c42bea5a9506d8a9bb23d0b2278724625d033f89" exitCode=137 Sep 08 14:36:36.091653 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:36.091233 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-37fa23" Sep 08 14:36:36.091653 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:36.091282 2578 scope.go:117] "RemoveContainer" containerID="c43d47817bd2b5c80b536c42c42bea5a9506d8a9bb23d0b2278724625d033f89" Sep 08 14:36:36.101049 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:36.101032 2578 scope.go:117] "RemoveContainer" containerID="c43d47817bd2b5c80b536c42c42bea5a9506d8a9bb23d0b2278724625d033f89" Sep 08 14:36:36.101347 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:36.101328 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c43d47817bd2b5c80b536c42c42bea5a9506d8a9bb23d0b2278724625d033f89\": container with ID starting with c43d47817bd2b5c80b536c42c42bea5a9506d8a9bb23d0b2278724625d033f89 not found: ID does not exist" containerID="c43d47817bd2b5c80b536c42c42bea5a9506d8a9bb23d0b2278724625d033f89" Sep 08 14:36:36.101428 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:36.101380 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c43d47817bd2b5c80b536c42c42bea5a9506d8a9bb23d0b2278724625d033f89"} err="failed to get container status \"c43d47817bd2b5c80b536c42c42bea5a9506d8a9bb23d0b2278724625d033f89\": rpc error: code = NotFound desc = could not find container \"c43d47817bd2b5c80b536c42c42bea5a9506d8a9bb23d0b2278724625d033f89\": container with ID starting with c43d47817bd2b5c80b536c42c42bea5a9506d8a9bb23d0b2278724625d033f89 not found: ID does not exist" Sep 08 14:36:36.101504 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:36.101473 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7895-37fa23\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="cd77d94e-3f5a-4aac-ac5b-1b26a195e78a" pod="opendatahub/test-curl-7895-37fa23" Sep 08 14:36:36.242246 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:36.242211 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd77d94e-3f5a-4aac-ac5b-1b26a195e78a" path="/var/lib/kubelet/pods/cd77d94e-3f5a-4aac-ac5b-1b26a195e78a/volumes" Sep 08 14:36:37.258758 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:37.258719 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-fdabeb" Sep 08 14:36:37.363313 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:37.363232 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnn69\" (UniqueName: \"kubernetes.io/projected/97c26fe5-d1e3-419d-9e76-7b529e33349e-kube-api-access-qnn69\") pod \"97c26fe5-d1e3-419d-9e76-7b529e33349e\" (UID: \"97c26fe5-d1e3-419d-9e76-7b529e33349e\") " Sep 08 14:36:37.365422 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:37.365394 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97c26fe5-d1e3-419d-9e76-7b529e33349e-kube-api-access-qnn69" (OuterVolumeSpecName: "kube-api-access-qnn69") pod "97c26fe5-d1e3-419d-9e76-7b529e33349e" (UID: "97c26fe5-d1e3-419d-9e76-7b529e33349e"). InnerVolumeSpecName "kube-api-access-qnn69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:36:37.464584 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:37.464551 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnn69\" (UniqueName: \"kubernetes.io/projected/97c26fe5-d1e3-419d-9e76-7b529e33349e-kube-api-access-qnn69\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:36:38.106393 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:38.106362 2578 generic.go:356] "Generic (PLEG): container finished" podID="97c26fe5-d1e3-419d-9e76-7b529e33349e" containerID="2e790461a0eba8ba9698fa37b30da7ef774b3726374a914c3229d52fcd2864b8" exitCode=137 Sep 08 14:36:38.106600 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:38.106403 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7895-fdabeb" Sep 08 14:36:38.106600 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:38.106463 2578 scope.go:117] "RemoveContainer" containerID="2e790461a0eba8ba9698fa37b30da7ef774b3726374a914c3229d52fcd2864b8" Sep 08 14:36:38.115747 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:38.115701 2578 scope.go:117] "RemoveContainer" containerID="2e790461a0eba8ba9698fa37b30da7ef774b3726374a914c3229d52fcd2864b8" Sep 08 14:36:38.115872 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:38.115840 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7895-fdabeb\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="97c26fe5-d1e3-419d-9e76-7b529e33349e" pod="opendatahub/test-curl-7895-fdabeb" Sep 08 14:36:38.115988 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:38.115968 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e790461a0eba8ba9698fa37b30da7ef774b3726374a914c3229d52fcd2864b8\": container with ID starting with 2e790461a0eba8ba9698fa37b30da7ef774b3726374a914c3229d52fcd2864b8 not found: ID does not exist" containerID="2e790461a0eba8ba9698fa37b30da7ef774b3726374a914c3229d52fcd2864b8" Sep 08 14:36:38.116031 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:38.115998 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e790461a0eba8ba9698fa37b30da7ef774b3726374a914c3229d52fcd2864b8"} err="failed to get container status \"2e790461a0eba8ba9698fa37b30da7ef774b3726374a914c3229d52fcd2864b8\": rpc error: code = NotFound desc = could not find container \"2e790461a0eba8ba9698fa37b30da7ef774b3726374a914c3229d52fcd2864b8\": container with ID starting with 2e790461a0eba8ba9698fa37b30da7ef774b3726374a914c3229d52fcd2864b8 not found: ID does not exist" Sep 08 14:36:38.241299 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:36:38.241257 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7895-fdabeb\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="97c26fe5-d1e3-419d-9e76-7b529e33349e" pod="opendatahub/test-curl-7895-fdabeb" Sep 08 14:36:38.243046 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:38.243017 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97c26fe5-d1e3-419d-9e76-7b529e33349e" path="/var/lib/kubelet/pods/97c26fe5-d1e3-419d-9e76-7b529e33349e/volumes" Sep 08 14:36:39.083527 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.083493 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc"] Sep 08 14:36:39.084102 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.084082 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97c26fe5-d1e3-419d-9e76-7b529e33349e" containerName="test-curl-7895-fdabeb" Sep 08 14:36:39.084189 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.084106 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="97c26fe5-d1e3-419d-9e76-7b529e33349e" containerName="test-curl-7895-fdabeb" Sep 08 14:36:39.084258 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.084243 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="97c26fe5-d1e3-419d-9e76-7b529e33349e" containerName="test-curl-7895-fdabeb" Sep 08 14:36:39.088757 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.088706 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.091099 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.091075 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-afb814fe-serving-cert\"" Sep 08 14:36:39.100456 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.100433 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc"] Sep 08 14:36:39.179463 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.179426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e287dad-2ff6-46f1-8deb-86d0b7232c33-metrics-certs\") pod \"maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.179622 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.179527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcgmb\" (UniqueName: \"kubernetes.io/projected/1e287dad-2ff6-46f1-8deb-86d0b7232c33-kube-api-access-hcgmb\") pod \"maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.179622 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.179554 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e287dad-2ff6-46f1-8deb-86d0b7232c33-maas-api-tls\") pod \"maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.179709 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.179639 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e287dad-2ff6-46f1-8deb-86d0b7232c33-openshift-service-ca\") pod \"maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.280646 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.280612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e287dad-2ff6-46f1-8deb-86d0b7232c33-openshift-service-ca\") pod \"maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.280830 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.280666 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e287dad-2ff6-46f1-8deb-86d0b7232c33-metrics-certs\") pod \"maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.280830 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.280713 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcgmb\" (UniqueName: \"kubernetes.io/projected/1e287dad-2ff6-46f1-8deb-86d0b7232c33-kube-api-access-hcgmb\") pod \"maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.280914 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.280854 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e287dad-2ff6-46f1-8deb-86d0b7232c33-maas-api-tls\") pod \"maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.281406 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.281315 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e287dad-2ff6-46f1-8deb-86d0b7232c33-openshift-service-ca\") pod \"maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.283213 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.283192 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e287dad-2ff6-46f1-8deb-86d0b7232c33-metrics-certs\") pod \"maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.283325 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.283252 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e287dad-2ff6-46f1-8deb-86d0b7232c33-maas-api-tls\") pod \"maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.292931 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.292908 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcgmb\" (UniqueName: \"kubernetes.io/projected/1e287dad-2ff6-46f1-8deb-86d0b7232c33-kube-api-access-hcgmb\") pod \"maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.401508 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.401427 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:39.680350 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:39.680126 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc"] Sep 08 14:36:39.683218 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:36:39.683189 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e287dad_2ff6_46f1_8deb_86d0b7232c33.slice/crio-0b1ffc92ae8953907d2a3af5f523a391800c321c93cc64dbd4c7e8caf7087d3f WatchSource:0}: Error finding container 0b1ffc92ae8953907d2a3af5f523a391800c321c93cc64dbd4c7e8caf7087d3f: Status 404 returned error can't find the container with id 0b1ffc92ae8953907d2a3af5f523a391800c321c93cc64dbd4c7e8caf7087d3f Sep 08 14:36:40.120656 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:40.120620 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" event={"ID":"1e287dad-2ff6-46f1-8deb-86d0b7232c33","Type":"ContainerStarted","Data":"0b1ffc92ae8953907d2a3af5f523a391800c321c93cc64dbd4c7e8caf7087d3f"} Sep 08 14:36:41.129296 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:41.129253 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" event={"ID":"1e287dad-2ff6-46f1-8deb-86d0b7232c33","Type":"ContainerStarted","Data":"d225fb4b2ffe6ec4af40f2c7e03d56d82bb0044dc1bf2e51e2ae708fe6842a00"} Sep 08 14:36:41.129837 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:41.129373 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:36:41.147167 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:41.147118 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" podStartSLOduration=1.607841101 podStartE2EDuration="2.147102577s" totalImagesPullingTime="539.261476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:36:41.145340149 +0000 UTC m=+1213.429657265" watchObservedRunningTime="2026-09-08 14:36:41.147102577 +0000 UTC m=+1213.431419688" Sep 08 14:36:47.140933 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:36:47.140897 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:37:53.194102 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.194065 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc"] Sep 08 14:37:53.195054 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.195017 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" podUID="1e287dad-2ff6-46f1-8deb-86d0b7232c33" containerName="maas-api" containerID="cri-o://d225fb4b2ffe6ec4af40f2c7e03d56d82bb0044dc1bf2e51e2ae708fe6842a00" gracePeriod=30 Sep 08 14:37:53.464013 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.463989 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:37:53.567847 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.567814 2578 generic.go:356] "Generic (PLEG): container finished" podID="1e287dad-2ff6-46f1-8deb-86d0b7232c33" containerID="d225fb4b2ffe6ec4af40f2c7e03d56d82bb0044dc1bf2e51e2ae708fe6842a00" exitCode=0 Sep 08 14:37:53.568001 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.567860 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" Sep 08 14:37:53.568001 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.567908 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" event={"ID":"1e287dad-2ff6-46f1-8deb-86d0b7232c33","Type":"ContainerDied","Data":"d225fb4b2ffe6ec4af40f2c7e03d56d82bb0044dc1bf2e51e2ae708fe6842a00"} Sep 08 14:37:53.568001 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.567964 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc" event={"ID":"1e287dad-2ff6-46f1-8deb-86d0b7232c33","Type":"ContainerDied","Data":"0b1ffc92ae8953907d2a3af5f523a391800c321c93cc64dbd4c7e8caf7087d3f"} Sep 08 14:37:53.568001 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.567986 2578 scope.go:117] "RemoveContainer" containerID="d225fb4b2ffe6ec4af40f2c7e03d56d82bb0044dc1bf2e51e2ae708fe6842a00" Sep 08 14:37:53.577506 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.577483 2578 scope.go:117] "RemoveContainer" containerID="d225fb4b2ffe6ec4af40f2c7e03d56d82bb0044dc1bf2e51e2ae708fe6842a00" Sep 08 14:37:53.577815 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:37:53.577787 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d225fb4b2ffe6ec4af40f2c7e03d56d82bb0044dc1bf2e51e2ae708fe6842a00\": container with ID starting with d225fb4b2ffe6ec4af40f2c7e03d56d82bb0044dc1bf2e51e2ae708fe6842a00 not found: ID does not exist" containerID="d225fb4b2ffe6ec4af40f2c7e03d56d82bb0044dc1bf2e51e2ae708fe6842a00" Sep 08 14:37:53.577872 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.577830 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d225fb4b2ffe6ec4af40f2c7e03d56d82bb0044dc1bf2e51e2ae708fe6842a00"} err="failed to get container status \"d225fb4b2ffe6ec4af40f2c7e03d56d82bb0044dc1bf2e51e2ae708fe6842a00\": rpc error: code = NotFound desc = could not find container \"d225fb4b2ffe6ec4af40f2c7e03d56d82bb0044dc1bf2e51e2ae708fe6842a00\": container with ID starting with d225fb4b2ffe6ec4af40f2c7e03d56d82bb0044dc1bf2e51e2ae708fe6842a00 not found: ID does not exist" Sep 08 14:37:53.593189 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.593155 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hcgmb\" (UniqueName: \"kubernetes.io/projected/1e287dad-2ff6-46f1-8deb-86d0b7232c33-kube-api-access-hcgmb\") pod \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " Sep 08 14:37:53.593357 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.593205 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e287dad-2ff6-46f1-8deb-86d0b7232c33-openshift-service-ca\") pod \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " Sep 08 14:37:53.593357 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.593292 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e287dad-2ff6-46f1-8deb-86d0b7232c33-metrics-certs\") pod \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " Sep 08 14:37:53.593505 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.593422 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e287dad-2ff6-46f1-8deb-86d0b7232c33-maas-api-tls\") pod \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\" (UID: \"1e287dad-2ff6-46f1-8deb-86d0b7232c33\") " Sep 08 14:37:53.593644 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.593609 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e287dad-2ff6-46f1-8deb-86d0b7232c33-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1e287dad-2ff6-46f1-8deb-86d0b7232c33" (UID: "1e287dad-2ff6-46f1-8deb-86d0b7232c33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:37:53.593837 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.593700 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e287dad-2ff6-46f1-8deb-86d0b7232c33-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:37:53.595793 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.595766 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e287dad-2ff6-46f1-8deb-86d0b7232c33-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1e287dad-2ff6-46f1-8deb-86d0b7232c33" (UID: "1e287dad-2ff6-46f1-8deb-86d0b7232c33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:37:53.595901 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.595827 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e287dad-2ff6-46f1-8deb-86d0b7232c33-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1e287dad-2ff6-46f1-8deb-86d0b7232c33" (UID: "1e287dad-2ff6-46f1-8deb-86d0b7232c33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:37:53.596105 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.596089 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e287dad-2ff6-46f1-8deb-86d0b7232c33-kube-api-access-hcgmb" (OuterVolumeSpecName: "kube-api-access-hcgmb") pod "1e287dad-2ff6-46f1-8deb-86d0b7232c33" (UID: "1e287dad-2ff6-46f1-8deb-86d0b7232c33"). InnerVolumeSpecName "kube-api-access-hcgmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:37:53.694175 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.694137 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e287dad-2ff6-46f1-8deb-86d0b7232c33-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:37:53.694175 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.694171 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcgmb\" (UniqueName: \"kubernetes.io/projected/1e287dad-2ff6-46f1-8deb-86d0b7232c33-kube-api-access-hcgmb\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:37:53.694175 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.694182 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e287dad-2ff6-46f1-8deb-86d0b7232c33-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:37:53.893368 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.893333 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc"] Sep 08 14:37:53.895883 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:53.895862 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-afb814fe-5bf57cc65c-znhwc"] Sep 08 14:37:54.244340 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:37:54.244292 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e287dad-2ff6-46f1-8deb-86d0b7232c33" path="/var/lib/kubelet/pods/1e287dad-2ff6-46f1-8deb-86d0b7232c33/volumes" Sep 08 14:38:57.849522 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:57.849439 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7898-f07226"] Sep 08 14:38:57.849932 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:57.849860 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e287dad-2ff6-46f1-8deb-86d0b7232c33" containerName="maas-api" Sep 08 14:38:57.849932 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:57.849872 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e287dad-2ff6-46f1-8deb-86d0b7232c33" containerName="maas-api" Sep 08 14:38:57.850006 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:57.849951 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e287dad-2ff6-46f1-8deb-86d0b7232c33" containerName="maas-api" Sep 08 14:38:57.858055 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:57.858022 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7898-f07226" Sep 08 14:38:57.860393 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:57.860367 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qcp6n\"" Sep 08 14:38:57.862992 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:57.862969 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7898-f07226"] Sep 08 14:38:57.976851 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:57.976810 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqvqt\" (UniqueName: \"kubernetes.io/projected/04dd2087-96f9-4c04-9535-6d1ad505ba9d-kube-api-access-dqvqt\") pod \"test-curl-7898-f07226\" (UID: \"04dd2087-96f9-4c04-9535-6d1ad505ba9d\") " pod="opendatahub/test-curl-7898-f07226" Sep 08 14:38:58.077507 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:58.077475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqvqt\" (UniqueName: \"kubernetes.io/projected/04dd2087-96f9-4c04-9535-6d1ad505ba9d-kube-api-access-dqvqt\") pod \"test-curl-7898-f07226\" (UID: \"04dd2087-96f9-4c04-9535-6d1ad505ba9d\") " pod="opendatahub/test-curl-7898-f07226" Sep 08 14:38:58.086764 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:58.086720 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqvqt\" (UniqueName: \"kubernetes.io/projected/04dd2087-96f9-4c04-9535-6d1ad505ba9d-kube-api-access-dqvqt\") pod \"test-curl-7898-f07226\" (UID: \"04dd2087-96f9-4c04-9535-6d1ad505ba9d\") " pod="opendatahub/test-curl-7898-f07226" Sep 08 14:38:58.169929 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:58.169829 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7898-f07226" Sep 08 14:38:58.439600 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:58.439577 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7898-f07226"] Sep 08 14:38:58.441510 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:38:58.441485 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04dd2087_96f9_4c04_9535_6d1ad505ba9d.slice/crio-02609daea0ccdba468d5e27a89b3933522386219dfb88a37ea3900bfc5db5404 WatchSource:0}: Error finding container 02609daea0ccdba468d5e27a89b3933522386219dfb88a37ea3900bfc5db5404: Status 404 returned error can't find the container with id 02609daea0ccdba468d5e27a89b3933522386219dfb88a37ea3900bfc5db5404 Sep 08 14:38:58.938248 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:58.938209 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7898-f07226" event={"ID":"04dd2087-96f9-4c04-9535-6d1ad505ba9d","Type":"ContainerStarted","Data":"d7e0bb0601c891dd0d55c13acdc4374f4565343f61db9e45b353232fc23f5c0d"} Sep 08 14:38:58.938248 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:58.938251 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7898-f07226" event={"ID":"04dd2087-96f9-4c04-9535-6d1ad505ba9d","Type":"ContainerStarted","Data":"02609daea0ccdba468d5e27a89b3933522386219dfb88a37ea3900bfc5db5404"} Sep 08 14:38:58.955032 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:58.954985 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7898-f07226" podStartSLOduration=1.954970865 podStartE2EDuration="1.954970865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:38:58.953642666 +0000 UTC m=+1351.237959784" watchObservedRunningTime="2026-09-08 14:38:58.954970865 +0000 UTC m=+1351.239288011" Sep 08 14:38:59.564601 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:59.564561 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7898-f07226"] Sep 08 14:38:59.573876 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:38:59.573653 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7898-f07226"] Sep 08 14:38:59.944022 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:38:59.943993 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7898-f07226" container="test-curl-7898-f07226" Sep 08 14:39:00.952059 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:00.952014 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7898-f07226" podUID="04dd2087-96f9-4c04-9535-6d1ad505ba9d" containerName="test-curl-7898-f07226" containerID="cri-o://d7e0bb0601c891dd0d55c13acdc4374f4565343f61db9e45b353232fc23f5c0d" gracePeriod=2 Sep 08 14:39:03.140080 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:03.140052 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7898-f07226" Sep 08 14:39:03.329870 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:03.329775 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqvqt\" (UniqueName: \"kubernetes.io/projected/04dd2087-96f9-4c04-9535-6d1ad505ba9d-kube-api-access-dqvqt\") pod \"04dd2087-96f9-4c04-9535-6d1ad505ba9d\" (UID: \"04dd2087-96f9-4c04-9535-6d1ad505ba9d\") " Sep 08 14:39:03.332481 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:03.332450 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04dd2087-96f9-4c04-9535-6d1ad505ba9d-kube-api-access-dqvqt" (OuterVolumeSpecName: "kube-api-access-dqvqt") pod "04dd2087-96f9-4c04-9535-6d1ad505ba9d" (UID: "04dd2087-96f9-4c04-9535-6d1ad505ba9d"). InnerVolumeSpecName "kube-api-access-dqvqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:39:03.431079 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:03.431042 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqvqt\" (UniqueName: \"kubernetes.io/projected/04dd2087-96f9-4c04-9535-6d1ad505ba9d-kube-api-access-dqvqt\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:39:03.969958 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:03.969921 2578 generic.go:356] "Generic (PLEG): container finished" podID="04dd2087-96f9-4c04-9535-6d1ad505ba9d" containerID="d7e0bb0601c891dd0d55c13acdc4374f4565343f61db9e45b353232fc23f5c0d" exitCode=137 Sep 08 14:39:03.970171 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:03.969970 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7898-f07226" Sep 08 14:39:03.970171 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:03.969987 2578 scope.go:117] "RemoveContainer" containerID="d7e0bb0601c891dd0d55c13acdc4374f4565343f61db9e45b353232fc23f5c0d" Sep 08 14:39:03.979377 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:03.979358 2578 scope.go:117] "RemoveContainer" containerID="d7e0bb0601c891dd0d55c13acdc4374f4565343f61db9e45b353232fc23f5c0d" Sep 08 14:39:03.979677 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:39:03.979656 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e0bb0601c891dd0d55c13acdc4374f4565343f61db9e45b353232fc23f5c0d\": container with ID starting with d7e0bb0601c891dd0d55c13acdc4374f4565343f61db9e45b353232fc23f5c0d not found: ID does not exist" containerID="d7e0bb0601c891dd0d55c13acdc4374f4565343f61db9e45b353232fc23f5c0d" Sep 08 14:39:03.979785 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:03.979685 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e0bb0601c891dd0d55c13acdc4374f4565343f61db9e45b353232fc23f5c0d"} err="failed to get container status \"d7e0bb0601c891dd0d55c13acdc4374f4565343f61db9e45b353232fc23f5c0d\": rpc error: code = NotFound desc = could not find container \"d7e0bb0601c891dd0d55c13acdc4374f4565343f61db9e45b353232fc23f5c0d\": container with ID starting with d7e0bb0601c891dd0d55c13acdc4374f4565343f61db9e45b353232fc23f5c0d not found: ID does not exist" Sep 08 14:39:03.979847 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:39:03.979777 2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7898-f07226\" is forbidden: User \"system:node:ip-10-0-141-155.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-155.ec2.internal' and this object" podUID="04dd2087-96f9-4c04-9535-6d1ad505ba9d" pod="opendatahub/test-curl-7898-f07226" Sep 08 14:39:04.242050 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:04.241969 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04dd2087-96f9-4c04-9535-6d1ad505ba9d" path="/var/lib/kubelet/pods/04dd2087-96f9-4c04-9535-6d1ad505ba9d/volumes" Sep 08 14:39:45.137743 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.137686 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl"] Sep 08 14:39:45.138379 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.138358 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04dd2087-96f9-4c04-9535-6d1ad505ba9d" containerName="test-curl-7898-f07226" Sep 08 14:39:45.138464 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.138382 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="04dd2087-96f9-4c04-9535-6d1ad505ba9d" containerName="test-curl-7898-f07226" Sep 08 14:39:45.138540 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.138526 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="04dd2087-96f9-4c04-9535-6d1ad505ba9d" containerName="test-curl-7898-f07226" Sep 08 14:39:45.143017 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.142995 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.145404 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.145379 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0fd47b-serving-cert\"" Sep 08 14:39:45.151434 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.151370 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl"] Sep 08 14:39:45.311348 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.311310 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c95977-dc3a-4ada-9831-af2f91376987-metrics-certs\") pod \"maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.311532 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.311446 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c95977-dc3a-4ada-9831-af2f91376987-maas-api-tls\") pod \"maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.311532 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.311488 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c95977-dc3a-4ada-9831-af2f91376987-openshift-service-ca\") pod \"maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.311611 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.311562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjqcz\" (UniqueName: \"kubernetes.io/projected/01c95977-dc3a-4ada-9831-af2f91376987-kube-api-access-rjqcz\") pod \"maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.412371 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.412274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c95977-dc3a-4ada-9831-af2f91376987-maas-api-tls\") pod \"maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.412371 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.412334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c95977-dc3a-4ada-9831-af2f91376987-openshift-service-ca\") pod \"maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.412607 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.412397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjqcz\" (UniqueName: \"kubernetes.io/projected/01c95977-dc3a-4ada-9831-af2f91376987-kube-api-access-rjqcz\") pod \"maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.412607 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.412428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c95977-dc3a-4ada-9831-af2f91376987-metrics-certs\") pod \"maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.413216 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.413188 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c95977-dc3a-4ada-9831-af2f91376987-openshift-service-ca\") pod \"maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.414930 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.414907 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c95977-dc3a-4ada-9831-af2f91376987-maas-api-tls\") pod \"maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.415020 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.414994 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c95977-dc3a-4ada-9831-af2f91376987-metrics-certs\") pod \"maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.421438 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.421418 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjqcz\" (UniqueName: \"kubernetes.io/projected/01c95977-dc3a-4ada-9831-af2f91376987-kube-api-access-rjqcz\") pod \"maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.458622 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.458595 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:45.734258 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.734229 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl"] Sep 08 14:39:45.735600 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:39:45.735577 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c95977_dc3a_4ada_9831_af2f91376987.slice/crio-6d5426947255901ee3454b242afaa2d9bcee2c7d15c0bfd7cc218355850141de WatchSource:0}: Error finding container 6d5426947255901ee3454b242afaa2d9bcee2c7d15c0bfd7cc218355850141de: Status 404 returned error can't find the container with id 6d5426947255901ee3454b242afaa2d9bcee2c7d15c0bfd7cc218355850141de Sep 08 14:39:45.737319 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:45.737304 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 14:39:46.208707 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:46.208670 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" event={"ID":"01c95977-dc3a-4ada-9831-af2f91376987","Type":"ContainerStarted","Data":"a85b3252b7b69a3fa516051a1054dab8b028ea6b0b3ae55bac5f0c4e5410648e"} Sep 08 14:39:46.209125 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:46.208741 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" event={"ID":"01c95977-dc3a-4ada-9831-af2f91376987","Type":"ContainerStarted","Data":"6d5426947255901ee3454b242afaa2d9bcee2c7d15c0bfd7cc218355850141de"} Sep 08 14:39:46.209125 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:46.208937 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:39:46.228786 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:46.228709 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" podStartSLOduration=0.83844124 podStartE2EDuration="1.228691751s" totalImagesPullingTime="390.250511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:39:46.22532451 +0000 UTC m=+1398.509641628" watchObservedRunningTime="2026-09-08 14:39:46.228691751 +0000 UTC m=+1398.513008866" Sep 08 14:39:52.219354 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:39:52.219327 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:40:14.119771 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.119712 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm"] Sep 08 14:40:14.123441 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.123417 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.125825 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.125805 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-41e512-serving-cert\"" Sep 08 14:40:14.134632 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.134605 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm"] Sep 08 14:40:14.176173 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.176142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-maas-api-tls\") pod \"maas-api-e2e-api-b-41e512-665859bff4-7snvm\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.176173 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.176178 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423d660a-e56c-49d2-a978-8a0643ca9eee-openshift-service-ca\") pod \"maas-api-e2e-api-b-41e512-665859bff4-7snvm\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.176434 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.176214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg5r6\" (UniqueName: \"kubernetes.io/projected/423d660a-e56c-49d2-a978-8a0643ca9eee-kube-api-access-rg5r6\") pod \"maas-api-e2e-api-b-41e512-665859bff4-7snvm\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.176434 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.176385 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-metrics-certs\") pod \"maas-api-e2e-api-b-41e512-665859bff4-7snvm\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.234211 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.234171 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7898-45492f"] Sep 08 14:40:14.238177 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.238153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7898-45492f" Sep 08 14:40:14.240802 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.240766 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qcp6n\"" Sep 08 14:40:14.246652 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.246197 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7898-45492f"] Sep 08 14:40:14.277397 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.277366 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-metrics-certs\") pod \"maas-api-e2e-api-b-41e512-665859bff4-7snvm\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.277588 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.277425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-maas-api-tls\") pod \"maas-api-e2e-api-b-41e512-665859bff4-7snvm\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.277588 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.277443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423d660a-e56c-49d2-a978-8a0643ca9eee-openshift-service-ca\") pod \"maas-api-e2e-api-b-41e512-665859bff4-7snvm\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.277588 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.277467 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg5r6\" (UniqueName: \"kubernetes.io/projected/423d660a-e56c-49d2-a978-8a0643ca9eee-kube-api-access-rg5r6\") pod \"maas-api-e2e-api-b-41e512-665859bff4-7snvm\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.277588 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:40:14.277574 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-serving-cert: secret "maas-api-e2e-api-b-41e512-serving-cert" not found Sep 08 14:40:14.277847 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:40:14.277645 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-maas-api-tls podName:423d660a-e56c-49d2-a978-8a0643ca9eee nodeName:}" failed. No retries permitted until 2026-09-08 14:40:14.777624846 +0000 UTC m=+1427.061941948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-maas-api-tls") pod "maas-api-e2e-api-b-41e512-665859bff4-7snvm" (UID: "423d660a-e56c-49d2-a978-8a0643ca9eee") : secret "maas-api-e2e-api-b-41e512-serving-cert" not found Sep 08 14:40:14.278165 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.278146 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423d660a-e56c-49d2-a978-8a0643ca9eee-openshift-service-ca\") pod \"maas-api-e2e-api-b-41e512-665859bff4-7snvm\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.280132 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.280108 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-metrics-certs\") pod \"maas-api-e2e-api-b-41e512-665859bff4-7snvm\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.288274 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.288248 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg5r6\" (UniqueName: \"kubernetes.io/projected/423d660a-e56c-49d2-a978-8a0643ca9eee-kube-api-access-rg5r6\") pod \"maas-api-e2e-api-b-41e512-665859bff4-7snvm\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.378469 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.378373 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qcr5\" (UniqueName: \"kubernetes.io/projected/f8efebe8-3810-479b-8982-80eb6a0618d5-kube-api-access-8qcr5\") pod \"test-curl-7898-45492f\" (UID: \"f8efebe8-3810-479b-8982-80eb6a0618d5\") " pod="opendatahub/test-curl-7898-45492f" Sep 08 14:40:14.479531 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.479496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qcr5\" (UniqueName: \"kubernetes.io/projected/f8efebe8-3810-479b-8982-80eb6a0618d5-kube-api-access-8qcr5\") pod \"test-curl-7898-45492f\" (UID: \"f8efebe8-3810-479b-8982-80eb6a0618d5\") " pod="opendatahub/test-curl-7898-45492f" Sep 08 14:40:14.487709 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.487673 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qcr5\" (UniqueName: \"kubernetes.io/projected/f8efebe8-3810-479b-8982-80eb6a0618d5-kube-api-access-8qcr5\") pod \"test-curl-7898-45492f\" (UID: \"f8efebe8-3810-479b-8982-80eb6a0618d5\") " pod="opendatahub/test-curl-7898-45492f" Sep 08 14:40:14.550097 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.550060 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7898-45492f" Sep 08 14:40:14.782863 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.782825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-maas-api-tls\") pod \"maas-api-e2e-api-b-41e512-665859bff4-7snvm\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.785701 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.785668 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-maas-api-tls\") pod \"maas-api-e2e-api-b-41e512-665859bff4-7snvm\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:14.824064 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:14.824038 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7898-45492f"] Sep 08 14:40:14.825535 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:40:14.825510 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8efebe8_3810_479b_8982_80eb6a0618d5.slice/crio-ab4f5337a5d711cd237d749d9d69a6c25f00486ff929a67254f8ef761b1f41da WatchSource:0}: Error finding container ab4f5337a5d711cd237d749d9d69a6c25f00486ff929a67254f8ef761b1f41da: Status 404 returned error can't find the container with id ab4f5337a5d711cd237d749d9d69a6c25f00486ff929a67254f8ef761b1f41da Sep 08 14:40:15.041360 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:15.041265 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:15.318513 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:15.318486 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm"] Sep 08 14:40:15.320177 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:40:15.320148 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423d660a_e56c_49d2_a978_8a0643ca9eee.slice/crio-0afdd454ca9ad3aded732dbc4175f996b8310ec86008795ee1344cf72c83ac43 WatchSource:0}: Error finding container 0afdd454ca9ad3aded732dbc4175f996b8310ec86008795ee1344cf72c83ac43: Status 404 returned error can't find the container with id 0afdd454ca9ad3aded732dbc4175f996b8310ec86008795ee1344cf72c83ac43 Sep 08 14:40:15.390564 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:15.390524 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7898-45492f" event={"ID":"f8efebe8-3810-479b-8982-80eb6a0618d5","Type":"ContainerStarted","Data":"94716559093a60a878a98630d4d58d5969a387e54aadef5ee7a047a1304faef4"} Sep 08 14:40:15.390767 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:15.390573 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7898-45492f" event={"ID":"f8efebe8-3810-479b-8982-80eb6a0618d5","Type":"ContainerStarted","Data":"ab4f5337a5d711cd237d749d9d69a6c25f00486ff929a67254f8ef761b1f41da"} Sep 08 14:40:15.392334 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:15.392300 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" event={"ID":"423d660a-e56c-49d2-a978-8a0643ca9eee","Type":"ContainerStarted","Data":"0afdd454ca9ad3aded732dbc4175f996b8310ec86008795ee1344cf72c83ac43"} Sep 08 14:40:15.405501 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:15.405454 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7898-45492f" podStartSLOduration=1.405439963 podStartE2EDuration="1.405439963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:40:15.403511008 +0000 UTC m=+1427.687828128" watchObservedRunningTime="2026-09-08 14:40:15.405439963 +0000 UTC m=+1427.689757120" Sep 08 14:40:16.009356 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.009321 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7898-45492f"] Sep 08 14:40:16.014579 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.014552 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7898-45492f"] Sep 08 14:40:16.132805 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.132709 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7898-5bfa0b"] Sep 08 14:40:16.133211 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.133195 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8efebe8-3810-479b-8982-80eb6a0618d5" containerName="test-curl-7898-45492f" Sep 08 14:40:16.133254 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.133215 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8efebe8-3810-479b-8982-80eb6a0618d5" containerName="test-curl-7898-45492f" Sep 08 14:40:16.133299 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.133290 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8efebe8-3810-479b-8982-80eb6a0618d5" containerName="test-curl-7898-45492f" Sep 08 14:40:16.136522 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.136505 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7898-5bfa0b" Sep 08 14:40:16.140584 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.140559 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7898-5bfa0b"] Sep 08 14:40:16.197104 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.197071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk55g\" (UniqueName: \"kubernetes.io/projected/4867543c-16ea-4b6d-86db-fbfc67dc0075-kube-api-access-kk55g\") pod \"test-curl-7898-5bfa0b\" (UID: \"4867543c-16ea-4b6d-86db-fbfc67dc0075\") " pod="opendatahub/test-curl-7898-5bfa0b" Sep 08 14:40:16.297962 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.297917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk55g\" (UniqueName: \"kubernetes.io/projected/4867543c-16ea-4b6d-86db-fbfc67dc0075-kube-api-access-kk55g\") pod \"test-curl-7898-5bfa0b\" (UID: \"4867543c-16ea-4b6d-86db-fbfc67dc0075\") " pod="opendatahub/test-curl-7898-5bfa0b" Sep 08 14:40:16.307496 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.307470 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk55g\" (UniqueName: \"kubernetes.io/projected/4867543c-16ea-4b6d-86db-fbfc67dc0075-kube-api-access-kk55g\") pod \"test-curl-7898-5bfa0b\" (UID: \"4867543c-16ea-4b6d-86db-fbfc67dc0075\") " pod="opendatahub/test-curl-7898-5bfa0b" Sep 08 14:40:16.400094 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.399995 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" event={"ID":"423d660a-e56c-49d2-a978-8a0643ca9eee","Type":"ContainerStarted","Data":"ab98d3e63aa0650aa6d706927594420dd7dd19b26add78b1d7aa138c1ea168f0"} Sep 08 14:40:16.400495 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.400214 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:16.400495 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:40:16.400231 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7898-45492f" container="test-curl-7898-45492f" Sep 08 14:40:16.416902 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.416844 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" podStartSLOduration=1.852677394 podStartE2EDuration="2.416828472s" totalImagesPullingTime="564.151078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:40:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:40:16.415433054 +0000 UTC m=+1428.699750191" watchObservedRunningTime="2026-09-08 14:40:16.416828472 +0000 UTC m=+1428.701145587" Sep 08 14:40:16.448139 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.448102 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7898-5bfa0b" Sep 08 14:40:16.720459 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:16.720232 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7898-5bfa0b"] Sep 08 14:40:16.722894 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:40:16.722866 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4867543c_16ea_4b6d_86db_fbfc67dc0075.slice/crio-c8b786e972a19571f4537967f3eb58b13ab00d89aa348c2124cff2ded9a19c1d WatchSource:0}: Error finding container c8b786e972a19571f4537967f3eb58b13ab00d89aa348c2124cff2ded9a19c1d: Status 404 returned error can't find the container with id c8b786e972a19571f4537967f3eb58b13ab00d89aa348c2124cff2ded9a19c1d Sep 08 14:40:17.407434 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:17.407392 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7898-5bfa0b" event={"ID":"4867543c-16ea-4b6d-86db-fbfc67dc0075","Type":"ContainerStarted","Data":"2d0dc194e8ba332b7de782396db1c77d20db1522281307d03bd90b75d8fa63f6"} Sep 08 14:40:17.407434 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:17.407437 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7898-5bfa0b" event={"ID":"4867543c-16ea-4b6d-86db-fbfc67dc0075","Type":"ContainerStarted","Data":"c8b786e972a19571f4537967f3eb58b13ab00d89aa348c2124cff2ded9a19c1d"} Sep 08 14:40:17.407968 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:17.407535 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7898-45492f" podUID="f8efebe8-3810-479b-8982-80eb6a0618d5" containerName="test-curl-7898-45492f" containerID="cri-o://94716559093a60a878a98630d4d58d5969a387e54aadef5ee7a047a1304faef4" gracePeriod=2 Sep 08 14:40:17.423154 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:17.423106 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7898-5bfa0b" podStartSLOduration=1.423091327 podStartE2EDuration="1.423091327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:40:17.421017532 +0000 UTC m=+1429.705334647" watchObservedRunningTime="2026-09-08 14:40:17.423091327 +0000 UTC m=+1429.707408468" Sep 08 14:40:18.036271 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:18.036238 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7898-5bfa0b"] Sep 08 14:40:18.039340 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:18.039308 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7898-5bfa0b"] Sep 08 14:40:18.412873 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:40:18.412846 2578 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7898-5bfa0b" container="test-curl-7898-5bfa0b" Sep 08 14:40:19.419385 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:19.419354 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7898-5bfa0b" podUID="4867543c-16ea-4b6d-86db-fbfc67dc0075" containerName="test-curl-7898-5bfa0b" containerID="cri-o://2d0dc194e8ba332b7de782396db1c77d20db1522281307d03bd90b75d8fa63f6" gracePeriod=2 Sep 08 14:40:19.600025 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:19.599999 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7898-45492f" Sep 08 14:40:19.733412 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:19.733380 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qcr5\" (UniqueName: \"kubernetes.io/projected/f8efebe8-3810-479b-8982-80eb6a0618d5-kube-api-access-8qcr5\") pod \"f8efebe8-3810-479b-8982-80eb6a0618d5\" (UID: \"f8efebe8-3810-479b-8982-80eb6a0618d5\") " Sep 08 14:40:19.735563 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:19.735542 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8efebe8-3810-479b-8982-80eb6a0618d5-kube-api-access-8qcr5" (OuterVolumeSpecName: "kube-api-access-8qcr5") pod "f8efebe8-3810-479b-8982-80eb6a0618d5" (UID: "f8efebe8-3810-479b-8982-80eb6a0618d5"). InnerVolumeSpecName "kube-api-access-8qcr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:40:19.834513 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:19.834476 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qcr5\" (UniqueName: \"kubernetes.io/projected/f8efebe8-3810-479b-8982-80eb6a0618d5-kube-api-access-8qcr5\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:40:20.242483 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:20.242451 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8efebe8-3810-479b-8982-80eb6a0618d5" path="/var/lib/kubelet/pods/f8efebe8-3810-479b-8982-80eb6a0618d5/volumes" Sep 08 14:40:20.426891 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:20.426861 2578 generic.go:356] "Generic (PLEG): container finished" podID="f8efebe8-3810-479b-8982-80eb6a0618d5" containerID="94716559093a60a878a98630d4d58d5969a387e54aadef5ee7a047a1304faef4" exitCode=137 Sep 08 14:40:20.427282 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:20.426908 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7898-45492f" Sep 08 14:40:20.427282 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:20.426966 2578 scope.go:117] "RemoveContainer" containerID="94716559093a60a878a98630d4d58d5969a387e54aadef5ee7a047a1304faef4" Sep 08 14:40:20.436014 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:20.435998 2578 scope.go:117] "RemoveContainer" containerID="94716559093a60a878a98630d4d58d5969a387e54aadef5ee7a047a1304faef4" Sep 08 14:40:20.436286 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:40:20.436257 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94716559093a60a878a98630d4d58d5969a387e54aadef5ee7a047a1304faef4\": container with ID starting with 94716559093a60a878a98630d4d58d5969a387e54aadef5ee7a047a1304faef4 not found: ID does not exist" containerID="94716559093a60a878a98630d4d58d5969a387e54aadef5ee7a047a1304faef4" Sep 08 14:40:20.436383 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:20.436291 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94716559093a60a878a98630d4d58d5969a387e54aadef5ee7a047a1304faef4"} err="failed to get container status \"94716559093a60a878a98630d4d58d5969a387e54aadef5ee7a047a1304faef4\": rpc error: code = NotFound desc = could not find container \"94716559093a60a878a98630d4d58d5969a387e54aadef5ee7a047a1304faef4\": container with ID starting with 94716559093a60a878a98630d4d58d5969a387e54aadef5ee7a047a1304faef4 not found: ID does not exist" Sep 08 14:40:21.564267 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:21.564239 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7898-5bfa0b" Sep 08 14:40:21.652347 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:21.652245 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk55g\" (UniqueName: \"kubernetes.io/projected/4867543c-16ea-4b6d-86db-fbfc67dc0075-kube-api-access-kk55g\") pod \"4867543c-16ea-4b6d-86db-fbfc67dc0075\" (UID: \"4867543c-16ea-4b6d-86db-fbfc67dc0075\") " Sep 08 14:40:21.654421 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:21.654389 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4867543c-16ea-4b6d-86db-fbfc67dc0075-kube-api-access-kk55g" (OuterVolumeSpecName: "kube-api-access-kk55g") pod "4867543c-16ea-4b6d-86db-fbfc67dc0075" (UID: "4867543c-16ea-4b6d-86db-fbfc67dc0075"). InnerVolumeSpecName "kube-api-access-kk55g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:40:21.753791 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:21.753723 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk55g\" (UniqueName: \"kubernetes.io/projected/4867543c-16ea-4b6d-86db-fbfc67dc0075-kube-api-access-kk55g\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:40:22.242839 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:22.242804 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4867543c-16ea-4b6d-86db-fbfc67dc0075" path="/var/lib/kubelet/pods/4867543c-16ea-4b6d-86db-fbfc67dc0075/volumes" Sep 08 14:40:22.413782 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:22.413750 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:40:22.440827 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:22.440781 2578 generic.go:356] "Generic (PLEG): container finished" podID="4867543c-16ea-4b6d-86db-fbfc67dc0075" containerID="2d0dc194e8ba332b7de782396db1c77d20db1522281307d03bd90b75d8fa63f6" exitCode=137 Sep 08 14:40:22.440827 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:22.440806 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7898-5bfa0b" Sep 08 14:40:22.441059 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:22.440851 2578 scope.go:117] "RemoveContainer" containerID="2d0dc194e8ba332b7de782396db1c77d20db1522281307d03bd90b75d8fa63f6" Sep 08 14:40:22.453413 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:22.453384 2578 scope.go:117] "RemoveContainer" containerID="2d0dc194e8ba332b7de782396db1c77d20db1522281307d03bd90b75d8fa63f6" Sep 08 14:40:22.453808 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:40:22.453780 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0dc194e8ba332b7de782396db1c77d20db1522281307d03bd90b75d8fa63f6\": container with ID starting with 2d0dc194e8ba332b7de782396db1c77d20db1522281307d03bd90b75d8fa63f6 not found: ID does not exist" containerID="2d0dc194e8ba332b7de782396db1c77d20db1522281307d03bd90b75d8fa63f6" Sep 08 14:40:22.453896 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:22.453819 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0dc194e8ba332b7de782396db1c77d20db1522281307d03bd90b75d8fa63f6"} err="failed to get container status \"2d0dc194e8ba332b7de782396db1c77d20db1522281307d03bd90b75d8fa63f6\": rpc error: code = NotFound desc = could not find container \"2d0dc194e8ba332b7de782396db1c77d20db1522281307d03bd90b75d8fa63f6\": container with ID starting with 2d0dc194e8ba332b7de782396db1c77d20db1522281307d03bd90b75d8fa63f6 not found: ID does not exist" Sep 08 14:40:56.663166 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:56.663135 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl"] Sep 08 14:40:56.663703 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:56.663350 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" podUID="01c95977-dc3a-4ada-9831-af2f91376987" containerName="maas-api" containerID="cri-o://a85b3252b7b69a3fa516051a1054dab8b028ea6b0b3ae55bac5f0c4e5410648e" gracePeriod=30 Sep 08 14:40:56.923065 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:56.923005 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:40:56.987888 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:56.987861 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjqcz\" (UniqueName: \"kubernetes.io/projected/01c95977-dc3a-4ada-9831-af2f91376987-kube-api-access-rjqcz\") pod \"01c95977-dc3a-4ada-9831-af2f91376987\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " Sep 08 14:40:56.988037 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:56.987916 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c95977-dc3a-4ada-9831-af2f91376987-metrics-certs\") pod \"01c95977-dc3a-4ada-9831-af2f91376987\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " Sep 08 14:40:56.988037 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:56.987953 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c95977-dc3a-4ada-9831-af2f91376987-openshift-service-ca\") pod \"01c95977-dc3a-4ada-9831-af2f91376987\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " Sep 08 14:40:56.988037 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:56.987995 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c95977-dc3a-4ada-9831-af2f91376987-maas-api-tls\") pod \"01c95977-dc3a-4ada-9831-af2f91376987\" (UID: \"01c95977-dc3a-4ada-9831-af2f91376987\") " Sep 08 14:40:56.988314 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:56.988281 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c95977-dc3a-4ada-9831-af2f91376987-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "01c95977-dc3a-4ada-9831-af2f91376987" (UID: "01c95977-dc3a-4ada-9831-af2f91376987"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:40:56.990224 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:56.990190 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c95977-dc3a-4ada-9831-af2f91376987-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "01c95977-dc3a-4ada-9831-af2f91376987" (UID: "01c95977-dc3a-4ada-9831-af2f91376987"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:40:56.990317 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:56.990234 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c95977-dc3a-4ada-9831-af2f91376987-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "01c95977-dc3a-4ada-9831-af2f91376987" (UID: "01c95977-dc3a-4ada-9831-af2f91376987"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:40:56.990317 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:56.990285 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c95977-dc3a-4ada-9831-af2f91376987-kube-api-access-rjqcz" (OuterVolumeSpecName: "kube-api-access-rjqcz") pod "01c95977-dc3a-4ada-9831-af2f91376987" (UID: "01c95977-dc3a-4ada-9831-af2f91376987"). InnerVolumeSpecName "kube-api-access-rjqcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:40:57.088855 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:57.088823 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjqcz\" (UniqueName: \"kubernetes.io/projected/01c95977-dc3a-4ada-9831-af2f91376987-kube-api-access-rjqcz\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:40:57.088855 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:57.088849 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01c95977-dc3a-4ada-9831-af2f91376987-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:40:57.088855 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:57.088858 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01c95977-dc3a-4ada-9831-af2f91376987-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:40:57.089069 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:57.088866 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01c95977-dc3a-4ada-9831-af2f91376987-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:40:57.646313 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:57.646277 2578 generic.go:356] "Generic (PLEG): container finished" podID="01c95977-dc3a-4ada-9831-af2f91376987" containerID="a85b3252b7b69a3fa516051a1054dab8b028ea6b0b3ae55bac5f0c4e5410648e" exitCode=0 Sep 08 14:40:57.646504 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:57.646340 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" Sep 08 14:40:57.646504 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:57.646370 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" event={"ID":"01c95977-dc3a-4ada-9831-af2f91376987","Type":"ContainerDied","Data":"a85b3252b7b69a3fa516051a1054dab8b028ea6b0b3ae55bac5f0c4e5410648e"} Sep 08 14:40:57.646504 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:57.646426 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl" event={"ID":"01c95977-dc3a-4ada-9831-af2f91376987","Type":"ContainerDied","Data":"6d5426947255901ee3454b242afaa2d9bcee2c7d15c0bfd7cc218355850141de"} Sep 08 14:40:57.646504 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:57.646448 2578 scope.go:117] "RemoveContainer" containerID="a85b3252b7b69a3fa516051a1054dab8b028ea6b0b3ae55bac5f0c4e5410648e" Sep 08 14:40:57.656487 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:57.656470 2578 scope.go:117] "RemoveContainer" containerID="a85b3252b7b69a3fa516051a1054dab8b028ea6b0b3ae55bac5f0c4e5410648e" Sep 08 14:40:57.656787 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:40:57.656766 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a85b3252b7b69a3fa516051a1054dab8b028ea6b0b3ae55bac5f0c4e5410648e\": container with ID starting with a85b3252b7b69a3fa516051a1054dab8b028ea6b0b3ae55bac5f0c4e5410648e not found: ID does not exist" containerID="a85b3252b7b69a3fa516051a1054dab8b028ea6b0b3ae55bac5f0c4e5410648e" Sep 08 14:40:57.656870 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:57.656794 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a85b3252b7b69a3fa516051a1054dab8b028ea6b0b3ae55bac5f0c4e5410648e"} err="failed to get container status \"a85b3252b7b69a3fa516051a1054dab8b028ea6b0b3ae55bac5f0c4e5410648e\": rpc error: code = NotFound desc = could not find container \"a85b3252b7b69a3fa516051a1054dab8b028ea6b0b3ae55bac5f0c4e5410648e\": container with ID starting with a85b3252b7b69a3fa516051a1054dab8b028ea6b0b3ae55bac5f0c4e5410648e not found: ID does not exist" Sep 08 14:40:57.668601 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:57.668579 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl"] Sep 08 14:40:57.673271 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:57.673254 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0fd47b-54f47fd988-n8fwl"] Sep 08 14:40:58.242537 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:40:58.242506 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01c95977-dc3a-4ada-9831-af2f91376987" path="/var/lib/kubelet/pods/01c95977-dc3a-4ada-9831-af2f91376987/volumes" Sep 08 14:41:19.318896 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.318859 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm"] Sep 08 14:41:19.319381 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.319114 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" podUID="423d660a-e56c-49d2-a978-8a0643ca9eee" containerName="maas-api" containerID="cri-o://ab98d3e63aa0650aa6d706927594420dd7dd19b26add78b1d7aa138c1ea168f0" gracePeriod=30 Sep 08 14:41:19.574280 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.574224 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:41:19.610067 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.609969 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg5r6\" (UniqueName: \"kubernetes.io/projected/423d660a-e56c-49d2-a978-8a0643ca9eee-kube-api-access-rg5r6\") pod \"423d660a-e56c-49d2-a978-8a0643ca9eee\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " Sep 08 14:41:19.610204 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.610128 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423d660a-e56c-49d2-a978-8a0643ca9eee-openshift-service-ca\") pod \"423d660a-e56c-49d2-a978-8a0643ca9eee\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " Sep 08 14:41:19.610204 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.610159 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-maas-api-tls\") pod \"423d660a-e56c-49d2-a978-8a0643ca9eee\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " Sep 08 14:41:19.610320 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.610211 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-metrics-certs\") pod \"423d660a-e56c-49d2-a978-8a0643ca9eee\" (UID: \"423d660a-e56c-49d2-a978-8a0643ca9eee\") " Sep 08 14:41:19.610521 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.610493 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423d660a-e56c-49d2-a978-8a0643ca9eee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "423d660a-e56c-49d2-a978-8a0643ca9eee" (UID: "423d660a-e56c-49d2-a978-8a0643ca9eee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:41:19.610629 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.610614 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423d660a-e56c-49d2-a978-8a0643ca9eee-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:41:19.612353 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.612328 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423d660a-e56c-49d2-a978-8a0643ca9eee-kube-api-access-rg5r6" (OuterVolumeSpecName: "kube-api-access-rg5r6") pod "423d660a-e56c-49d2-a978-8a0643ca9eee" (UID: "423d660a-e56c-49d2-a978-8a0643ca9eee"). InnerVolumeSpecName "kube-api-access-rg5r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:41:19.612829 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.612808 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "423d660a-e56c-49d2-a978-8a0643ca9eee" (UID: "423d660a-e56c-49d2-a978-8a0643ca9eee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:41:19.612938 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.612856 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "423d660a-e56c-49d2-a978-8a0643ca9eee" (UID: "423d660a-e56c-49d2-a978-8a0643ca9eee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:41:19.711567 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.711526 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg5r6\" (UniqueName: \"kubernetes.io/projected/423d660a-e56c-49d2-a978-8a0643ca9eee-kube-api-access-rg5r6\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:41:19.711567 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.711565 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:41:19.711567 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.711575 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/423d660a-e56c-49d2-a978-8a0643ca9eee-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:41:19.781073 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.781042 2578 generic.go:356] "Generic (PLEG): container finished" podID="423d660a-e56c-49d2-a978-8a0643ca9eee" containerID="ab98d3e63aa0650aa6d706927594420dd7dd19b26add78b1d7aa138c1ea168f0" exitCode=0 Sep 08 14:41:19.781236 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.781093 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" Sep 08 14:41:19.781236 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.781127 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" event={"ID":"423d660a-e56c-49d2-a978-8a0643ca9eee","Type":"ContainerDied","Data":"ab98d3e63aa0650aa6d706927594420dd7dd19b26add78b1d7aa138c1ea168f0"} Sep 08 14:41:19.781236 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.781183 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm" event={"ID":"423d660a-e56c-49d2-a978-8a0643ca9eee","Type":"ContainerDied","Data":"0afdd454ca9ad3aded732dbc4175f996b8310ec86008795ee1344cf72c83ac43"} Sep 08 14:41:19.781236 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.781204 2578 scope.go:117] "RemoveContainer" containerID="ab98d3e63aa0650aa6d706927594420dd7dd19b26add78b1d7aa138c1ea168f0" Sep 08 14:41:19.791458 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.791443 2578 scope.go:117] "RemoveContainer" containerID="ab98d3e63aa0650aa6d706927594420dd7dd19b26add78b1d7aa138c1ea168f0" Sep 08 14:41:19.791802 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:41:19.791778 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab98d3e63aa0650aa6d706927594420dd7dd19b26add78b1d7aa138c1ea168f0\": container with ID starting with ab98d3e63aa0650aa6d706927594420dd7dd19b26add78b1d7aa138c1ea168f0 not found: ID does not exist" containerID="ab98d3e63aa0650aa6d706927594420dd7dd19b26add78b1d7aa138c1ea168f0" Sep 08 14:41:19.791915 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.791809 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab98d3e63aa0650aa6d706927594420dd7dd19b26add78b1d7aa138c1ea168f0"} err="failed to get container status \"ab98d3e63aa0650aa6d706927594420dd7dd19b26add78b1d7aa138c1ea168f0\": rpc error: code = NotFound desc = could not find container \"ab98d3e63aa0650aa6d706927594420dd7dd19b26add78b1d7aa138c1ea168f0\": container with ID starting with ab98d3e63aa0650aa6d706927594420dd7dd19b26add78b1d7aa138c1ea168f0 not found: ID does not exist" Sep 08 14:41:19.803341 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.803320 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm"] Sep 08 14:41:19.807806 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:19.807787 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-41e512-665859bff4-7snvm"] Sep 08 14:41:20.246754 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:20.244032 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="423d660a-e56c-49d2-a978-8a0643ca9eee" path="/var/lib/kubelet/pods/423d660a-e56c-49d2-a978-8a0643ca9eee/volumes" Sep 08 14:41:28.290255 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:28.290228 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:41:28.290840 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:28.290824 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:41:28.298568 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:28.298547 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:41:28.299139 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:28.299112 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:41:37.343667 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:37.343633 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm"] Sep 08 14:41:37.369994 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:41:37.369967 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-serving-cert: secret "maas-api-e2e-shared-a-w2-23a1d4-serving-cert" not found Sep 08 14:41:37.370162 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:41:37.370041 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-maas-api-tls podName:393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285 nodeName:}" failed. No retries permitted until 2026-09-08 14:41:37.870021179 +0000 UTC m=+1510.154338292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-maas-api-tls") pod "maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" (UID: "393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285") : secret "maas-api-e2e-shared-a-w2-23a1d4-serving-cert" not found Sep 08 14:41:37.874371 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:41:37.874336 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-serving-cert: secret "maas-api-e2e-shared-a-w2-23a1d4-serving-cert" not found Sep 08 14:41:37.874543 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:41:37.874400 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-maas-api-tls podName:393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285 nodeName:}" failed. No retries permitted until 2026-09-08 14:41:38.87438714 +0000 UTC m=+1511.158704233 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-maas-api-tls") pod "maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" (UID: "393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285") : secret "maas-api-e2e-shared-a-w2-23a1d4-serving-cert" not found Sep 08 14:41:37.879954 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:37.879901 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" podUID="393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285" containerName="maas-api" containerID="cri-o://7427d90d17bdf35f7cb252bbbbd1ef6390b41fcf6ae95205e8507dae86e2e6a8" gracePeriod=30 Sep 08 14:41:38.136832 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.136774 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:41:38.177375 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.177343 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-metrics-certs\") pod \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " Sep 08 14:41:38.177537 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.177389 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-openshift-service-ca\") pod \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " Sep 08 14:41:38.177639 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.177617 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-maas-api-tls\") pod \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " Sep 08 14:41:38.177712 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.177688 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbhs6\" (UniqueName: \"kubernetes.io/projected/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-kube-api-access-cbhs6\") pod \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\" (UID: \"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285\") " Sep 08 14:41:38.177804 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.177774 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285" (UID: "393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:41:38.178107 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.178083 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:41:38.179904 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.179874 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-kube-api-access-cbhs6" (OuterVolumeSpecName: "kube-api-access-cbhs6") pod "393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285" (UID: "393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285"). InnerVolumeSpecName "kube-api-access-cbhs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:41:38.180009 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.179936 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285" (UID: "393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:41:38.180059 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.180046 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285" (UID: "393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:41:38.278842 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.278814 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:41:38.278842 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.278842 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbhs6\" (UniqueName: \"kubernetes.io/projected/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-kube-api-access-cbhs6\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:41:38.279001 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.278855 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:41:38.886584 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.886545 2578 generic.go:356] "Generic (PLEG): container finished" podID="393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285" containerID="7427d90d17bdf35f7cb252bbbbd1ef6390b41fcf6ae95205e8507dae86e2e6a8" exitCode=0 Sep 08 14:41:38.887023 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.886593 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" event={"ID":"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285","Type":"ContainerDied","Data":"7427d90d17bdf35f7cb252bbbbd1ef6390b41fcf6ae95205e8507dae86e2e6a8"} Sep 08 14:41:38.887023 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.886604 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" Sep 08 14:41:38.887023 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.886645 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm" event={"ID":"393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285","Type":"ContainerDied","Data":"be70896b14ce848a065d594971c55e30e286a634e2000f774c8b79b639193db0"} Sep 08 14:41:38.887023 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.886668 2578 scope.go:117] "RemoveContainer" containerID="7427d90d17bdf35f7cb252bbbbd1ef6390b41fcf6ae95205e8507dae86e2e6a8" Sep 08 14:41:38.896033 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.896016 2578 scope.go:117] "RemoveContainer" containerID="7427d90d17bdf35f7cb252bbbbd1ef6390b41fcf6ae95205e8507dae86e2e6a8" Sep 08 14:41:38.896296 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:41:38.896276 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7427d90d17bdf35f7cb252bbbbd1ef6390b41fcf6ae95205e8507dae86e2e6a8\": container with ID starting with 7427d90d17bdf35f7cb252bbbbd1ef6390b41fcf6ae95205e8507dae86e2e6a8 not found: ID does not exist" containerID="7427d90d17bdf35f7cb252bbbbd1ef6390b41fcf6ae95205e8507dae86e2e6a8" Sep 08 14:41:38.896347 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.896304 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7427d90d17bdf35f7cb252bbbbd1ef6390b41fcf6ae95205e8507dae86e2e6a8"} err="failed to get container status \"7427d90d17bdf35f7cb252bbbbd1ef6390b41fcf6ae95205e8507dae86e2e6a8\": rpc error: code = NotFound desc = could not find container \"7427d90d17bdf35f7cb252bbbbd1ef6390b41fcf6ae95205e8507dae86e2e6a8\": container with ID starting with 7427d90d17bdf35f7cb252bbbbd1ef6390b41fcf6ae95205e8507dae86e2e6a8 not found: ID does not exist" Sep 08 14:41:38.905326 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.905304 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm"] Sep 08 14:41:38.909116 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:38.909098 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-23a1d4-5bf569d8ff-mtjgm"] Sep 08 14:41:40.242491 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:40.242460 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285" path="/var/lib/kubelet/pods/393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285/volumes" Sep 08 14:41:55.455462 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.455367 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98"] Sep 08 14:41:55.455944 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.455641 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" podUID="ff62c013-0f6e-461e-9f8a-90faed677687" containerName="maas-api" containerID="cri-o://131c040b7386c8676f798b0fa411890aa23e78f3ada17c78415c44b279bc8c2d" gracePeriod=30 Sep 08 14:41:55.720109 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.720038 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:41:55.843126 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843096 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l"] Sep 08 14:41:55.843496 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843484 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff62c013-0f6e-461e-9f8a-90faed677687" containerName="maas-api" Sep 08 14:41:55.843544 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843498 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff62c013-0f6e-461e-9f8a-90faed677687" containerName="maas-api" Sep 08 14:41:55.843544 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843514 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423d660a-e56c-49d2-a978-8a0643ca9eee" containerName="maas-api" Sep 08 14:41:55.843544 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843519 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="423d660a-e56c-49d2-a978-8a0643ca9eee" containerName="maas-api" Sep 08 14:41:55.843544 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843529 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285" containerName="maas-api" Sep 08 14:41:55.843544 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843534 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285" containerName="maas-api" Sep 08 14:41:55.843701 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843548 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01c95977-dc3a-4ada-9831-af2f91376987" containerName="maas-api" Sep 08 14:41:55.843701 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843554 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="01c95977-dc3a-4ada-9831-af2f91376987" containerName="maas-api" Sep 08 14:41:55.843701 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843559 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4867543c-16ea-4b6d-86db-fbfc67dc0075" containerName="test-curl-7898-5bfa0b" Sep 08 14:41:55.843701 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843565 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4867543c-16ea-4b6d-86db-fbfc67dc0075" containerName="test-curl-7898-5bfa0b" Sep 08 14:41:55.843701 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843621 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="423d660a-e56c-49d2-a978-8a0643ca9eee" containerName="maas-api" Sep 08 14:41:55.843701 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843628 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4867543c-16ea-4b6d-86db-fbfc67dc0075" containerName="test-curl-7898-5bfa0b" Sep 08 14:41:55.843701 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843636 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="393fc1d9-bbe5-4c7b-a63b-8e1b3ccfe285" containerName="maas-api" Sep 08 14:41:55.843701 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843641 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="01c95977-dc3a-4ada-9831-af2f91376987" containerName="maas-api" Sep 08 14:41:55.843701 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.843647 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff62c013-0f6e-461e-9f8a-90faed677687" containerName="maas-api" Sep 08 14:41:55.846814 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.846798 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:55.849213 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.849189 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-274ffa-serving-cert\"" Sep 08 14:41:55.856494 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.856473 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l"] Sep 08 14:41:55.857079 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.857056 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-maas-api-tls\") pod \"ff62c013-0f6e-461e-9f8a-90faed677687\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " Sep 08 14:41:55.857199 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.857130 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlmtw\" (UniqueName: \"kubernetes.io/projected/ff62c013-0f6e-461e-9f8a-90faed677687-kube-api-access-hlmtw\") pod \"ff62c013-0f6e-461e-9f8a-90faed677687\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " Sep 08 14:41:55.857199 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.857157 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-metrics-certs\") pod \"ff62c013-0f6e-461e-9f8a-90faed677687\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " Sep 08 14:41:55.857312 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.857274 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff62c013-0f6e-461e-9f8a-90faed677687-openshift-service-ca\") pod \"ff62c013-0f6e-461e-9f8a-90faed677687\" (UID: \"ff62c013-0f6e-461e-9f8a-90faed677687\") " Sep 08 14:41:55.857989 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.857958 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff62c013-0f6e-461e-9f8a-90faed677687-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ff62c013-0f6e-461e-9f8a-90faed677687" (UID: "ff62c013-0f6e-461e-9f8a-90faed677687"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:41:55.860991 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.860964 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ff62c013-0f6e-461e-9f8a-90faed677687" (UID: "ff62c013-0f6e-461e-9f8a-90faed677687"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:41:55.860991 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.860977 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff62c013-0f6e-461e-9f8a-90faed677687-kube-api-access-hlmtw" (OuterVolumeSpecName: "kube-api-access-hlmtw") pod "ff62c013-0f6e-461e-9f8a-90faed677687" (UID: "ff62c013-0f6e-461e-9f8a-90faed677687"). InnerVolumeSpecName "kube-api-access-hlmtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:41:55.861157 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.861033 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ff62c013-0f6e-461e-9f8a-90faed677687" (UID: "ff62c013-0f6e-461e-9f8a-90faed677687"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:41:55.958583 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.958551 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9813e7de-1f89-4b70-bff6-891edd0704e3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:55.958797 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.958619 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdss5\" (UniqueName: \"kubernetes.io/projected/9813e7de-1f89-4b70-bff6-891edd0704e3-kube-api-access-fdss5\") pod \"maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:55.958797 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.958741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:55.958797 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.958773 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-metrics-certs\") pod \"maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:55.958921 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.958861 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff62c013-0f6e-461e-9f8a-90faed677687-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:41:55.958921 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.958872 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:41:55.958921 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.958881 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlmtw\" (UniqueName: \"kubernetes.io/projected/ff62c013-0f6e-461e-9f8a-90faed677687-kube-api-access-hlmtw\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:41:55.958921 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.958889 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff62c013-0f6e-461e-9f8a-90faed677687-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:41:55.986616 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.986521 2578 generic.go:356] "Generic (PLEG): container finished" podID="ff62c013-0f6e-461e-9f8a-90faed677687" containerID="131c040b7386c8676f798b0fa411890aa23e78f3ada17c78415c44b279bc8c2d" exitCode=0 Sep 08 14:41:55.986616 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.986570 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" Sep 08 14:41:55.986616 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.986577 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" event={"ID":"ff62c013-0f6e-461e-9f8a-90faed677687","Type":"ContainerDied","Data":"131c040b7386c8676f798b0fa411890aa23e78f3ada17c78415c44b279bc8c2d"} Sep 08 14:41:55.986905 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.986630 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98" event={"ID":"ff62c013-0f6e-461e-9f8a-90faed677687","Type":"ContainerDied","Data":"9844cc327ce9f161e21d2d2017b8c11afe966c830e8a276621b0fca631403db1"} Sep 08 14:41:55.986905 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.986660 2578 scope.go:117] "RemoveContainer" containerID="131c040b7386c8676f798b0fa411890aa23e78f3ada17c78415c44b279bc8c2d" Sep 08 14:41:55.997599 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.997580 2578 scope.go:117] "RemoveContainer" containerID="131c040b7386c8676f798b0fa411890aa23e78f3ada17c78415c44b279bc8c2d" Sep 08 14:41:55.997919 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:41:55.997899 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131c040b7386c8676f798b0fa411890aa23e78f3ada17c78415c44b279bc8c2d\": container with ID starting with 131c040b7386c8676f798b0fa411890aa23e78f3ada17c78415c44b279bc8c2d not found: ID does not exist" containerID="131c040b7386c8676f798b0fa411890aa23e78f3ada17c78415c44b279bc8c2d" Sep 08 14:41:55.997976 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:55.997926 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131c040b7386c8676f798b0fa411890aa23e78f3ada17c78415c44b279bc8c2d"} err="failed to get container status \"131c040b7386c8676f798b0fa411890aa23e78f3ada17c78415c44b279bc8c2d\": rpc error: code = NotFound desc = could not find container \"131c040b7386c8676f798b0fa411890aa23e78f3ada17c78415c44b279bc8c2d\": container with ID starting with 131c040b7386c8676f798b0fa411890aa23e78f3ada17c78415c44b279bc8c2d not found: ID does not exist" Sep 08 14:41:56.009834 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:56.009796 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98"] Sep 08 14:41:56.015412 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:56.015388 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-34656b-9c88c8bb5-btq98"] Sep 08 14:41:56.059892 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:56.059865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdss5\" (UniqueName: \"kubernetes.io/projected/9813e7de-1f89-4b70-bff6-891edd0704e3-kube-api-access-fdss5\") pod \"maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:56.059992 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:56.059937 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:56.059992 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:56.059966 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-metrics-certs\") pod \"maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:56.060113 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:56.060026 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9813e7de-1f89-4b70-bff6-891edd0704e3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:56.060181 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:41:56.060110 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-serving-cert: secret "maas-api-e2e-ipp-a-274ffa-serving-cert" not found Sep 08 14:41:56.060230 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:41:56.060190 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-maas-api-tls podName:9813e7de-1f89-4b70-bff6-891edd0704e3 nodeName:}" failed. No retries permitted until 2026-09-08 14:41:56.560168077 +0000 UTC m=+1528.844485186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-maas-api-tls") pod "maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" (UID: "9813e7de-1f89-4b70-bff6-891edd0704e3") : secret "maas-api-e2e-ipp-a-274ffa-serving-cert" not found Sep 08 14:41:56.060639 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:56.060621 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9813e7de-1f89-4b70-bff6-891edd0704e3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:56.062481 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:56.062459 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-metrics-certs\") pod \"maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:56.070850 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:56.070831 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdss5\" (UniqueName: \"kubernetes.io/projected/9813e7de-1f89-4b70-bff6-891edd0704e3-kube-api-access-fdss5\") pod \"maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:56.242749 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:56.242655 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff62c013-0f6e-461e-9f8a-90faed677687" path="/var/lib/kubelet/pods/ff62c013-0f6e-461e-9f8a-90faed677687/volumes" Sep 08 14:41:56.564797 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:56.564688 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:56.567396 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:56.567371 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:56.761578 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:56.761547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:57.039957 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:57.039929 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l"] Sep 08 14:41:57.041602 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:41:57.041575 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9813e7de_1f89_4b70_bff6_891edd0704e3.slice/crio-f8bbca793aa22e64213f28678042aeff5149fefa8011c24d70d90dd03eda7d75 WatchSource:0}: Error finding container f8bbca793aa22e64213f28678042aeff5149fefa8011c24d70d90dd03eda7d75: Status 404 returned error can't find the container with id f8bbca793aa22e64213f28678042aeff5149fefa8011c24d70d90dd03eda7d75 Sep 08 14:41:58.002691 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:58.002659 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" event={"ID":"9813e7de-1f89-4b70-bff6-891edd0704e3","Type":"ContainerStarted","Data":"72544ade6d098e55cd8e309962ae08d790457d76d01c550ff0ab2d4665cf6069"} Sep 08 14:41:58.003105 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:58.002708 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" event={"ID":"9813e7de-1f89-4b70-bff6-891edd0704e3","Type":"ContainerStarted","Data":"f8bbca793aa22e64213f28678042aeff5149fefa8011c24d70d90dd03eda7d75"} Sep 08 14:41:58.003105 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:58.002770 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:41:58.025677 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:41:58.025624 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" podStartSLOduration=2.529493497 podStartE2EDuration="3.025611429s" totalImagesPullingTime="496.117932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:41:58.024025678 +0000 UTC m=+1530.308342795" watchObservedRunningTime="2026-09-08 14:41:58.025611429 +0000 UTC m=+1530.309928544" Sep 08 14:42:04.013691 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:04.013657 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:42:24.941404 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:24.941364 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288"] Sep 08 14:42:24.945618 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:24.945599 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:24.947945 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:24.947921 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f7d00a-serving-cert\"" Sep 08 14:42:24.953221 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:24.953199 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288"] Sep 08 14:42:25.010142 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.010109 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab8c321-5664-4047-824b-00bfc7501c47-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:25.010255 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.010150 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2ms\" (UniqueName: \"kubernetes.io/projected/7ab8c321-5664-4047-824b-00bfc7501c47-kube-api-access-ws2ms\") pod \"maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:25.010255 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.010177 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab8c321-5664-4047-824b-00bfc7501c47-metrics-certs\") pod \"maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:25.010333 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.010270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8c321-5664-4047-824b-00bfc7501c47-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:25.111609 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.111576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab8c321-5664-4047-824b-00bfc7501c47-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:25.111819 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.111622 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2ms\" (UniqueName: \"kubernetes.io/projected/7ab8c321-5664-4047-824b-00bfc7501c47-kube-api-access-ws2ms\") pod \"maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:25.111819 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.111653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab8c321-5664-4047-824b-00bfc7501c47-metrics-certs\") pod \"maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:25.111819 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.111694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8c321-5664-4047-824b-00bfc7501c47-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:25.112405 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.112385 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8c321-5664-4047-824b-00bfc7501c47-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:25.114204 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.114180 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab8c321-5664-4047-824b-00bfc7501c47-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:25.114344 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.114325 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab8c321-5664-4047-824b-00bfc7501c47-metrics-certs\") pod \"maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:25.120751 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.120664 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2ms\" (UniqueName: \"kubernetes.io/projected/7ab8c321-5664-4047-824b-00bfc7501c47-kube-api-access-ws2ms\") pod \"maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:25.259381 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.259349 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:25.534410 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:25.534387 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288"] Sep 08 14:42:25.536689 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:42:25.536661 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab8c321_5664_4047_824b_00bfc7501c47.slice/crio-fc8b3d19ff1b66a4efe25c7110c4ca51e818b2b420db8a4aab50a6d875481a5b WatchSource:0}: Error finding container fc8b3d19ff1b66a4efe25c7110c4ca51e818b2b420db8a4aab50a6d875481a5b: Status 404 returned error can't find the container with id fc8b3d19ff1b66a4efe25c7110c4ca51e818b2b420db8a4aab50a6d875481a5b Sep 08 14:42:26.159381 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:26.159298 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" event={"ID":"7ab8c321-5664-4047-824b-00bfc7501c47","Type":"ContainerStarted","Data":"a112e201e940d109a0700e33464b1eee365daffb98795d78ca0ea0cbe9e5adbd"} Sep 08 14:42:26.159381 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:26.159356 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" event={"ID":"7ab8c321-5664-4047-824b-00bfc7501c47","Type":"ContainerStarted","Data":"fc8b3d19ff1b66a4efe25c7110c4ca51e818b2b420db8a4aab50a6d875481a5b"} Sep 08 14:42:26.159795 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:26.159512 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:26.178914 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:26.178863 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" podStartSLOduration=1.8461658189999999 podStartE2EDuration="2.178848833s" totalImagesPullingTime="332.683014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:42:26.176333217 +0000 UTC m=+1558.460650335" watchObservedRunningTime="2026-09-08 14:42:26.178848833 +0000 UTC m=+1558.463165949" Sep 08 14:42:32.174050 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:32.174021 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:42:53.905378 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:42:53.905346 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9xh8f_88bb47cd-a680-4da8-8e34-092fb40d590d/payload-processing/0.log" Sep 08 14:43:31.974582 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:31.974508 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9xh8f_88bb47cd-a680-4da8-8e34-092fb40d590d/payload-processing/0.log" Sep 08 14:43:37.356899 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.356868 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj"] Sep 08 14:43:37.360632 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.360615 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:37.363159 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.363134 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-6b7995-serving-cert\"" Sep 08 14:43:37.372505 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.372483 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj"] Sep 08 14:43:37.395136 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.395108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:37.395263 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.395152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:37.395263 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.395229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtl6g\" (UniqueName: \"kubernetes.io/projected/14af086a-60d8-4e7e-b501-eb17f2de538d-kube-api-access-rtl6g\") pod \"maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:37.395343 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.395263 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14af086a-60d8-4e7e-b501-eb17f2de538d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:37.496669 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.496633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:37.496888 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.496683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtl6g\" (UniqueName: \"kubernetes.io/projected/14af086a-60d8-4e7e-b501-eb17f2de538d-kube-api-access-rtl6g\") pod \"maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:37.496888 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.496705 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14af086a-60d8-4e7e-b501-eb17f2de538d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:37.496888 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.496802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:37.496888 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:43:37.496823 2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-serving-cert: secret "maas-api-e2e-ipp-cleanup-6b7995-serving-cert" not found Sep 08 14:43:37.497123 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:43:37.496908 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-maas-api-tls podName:14af086a-60d8-4e7e-b501-eb17f2de538d nodeName:}" failed. No retries permitted until 2026-09-08 14:43:37.996887398 +0000 UTC m=+1630.281204491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" (UID: "14af086a-60d8-4e7e-b501-eb17f2de538d") : secret "maas-api-e2e-ipp-cleanup-6b7995-serving-cert" not found Sep 08 14:43:37.497387 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.497365 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14af086a-60d8-4e7e-b501-eb17f2de538d-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:37.499307 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.499282 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:37.505142 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:37.505121 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtl6g\" (UniqueName: \"kubernetes.io/projected/14af086a-60d8-4e7e-b501-eb17f2de538d-kube-api-access-rtl6g\") pod \"maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:38.002228 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:38.002195 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:38.004937 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:38.004913 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:38.273501 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:38.273414 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:38.545763 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:38.545721 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj"] Sep 08 14:43:38.547457 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:43:38.547432 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14af086a_60d8_4e7e_b501_eb17f2de538d.slice/crio-c9e6dcbd9912bfd15450ccf7ca07546e6153dc3f282324d2b1d4d276493b6d2b WatchSource:0}: Error finding container c9e6dcbd9912bfd15450ccf7ca07546e6153dc3f282324d2b1d4d276493b6d2b: Status 404 returned error can't find the container with id c9e6dcbd9912bfd15450ccf7ca07546e6153dc3f282324d2b1d4d276493b6d2b Sep 08 14:43:38.566781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:38.566749 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" event={"ID":"14af086a-60d8-4e7e-b501-eb17f2de538d","Type":"ContainerStarted","Data":"c9e6dcbd9912bfd15450ccf7ca07546e6153dc3f282324d2b1d4d276493b6d2b"} Sep 08 14:43:39.574164 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:39.574130 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" event={"ID":"14af086a-60d8-4e7e-b501-eb17f2de538d","Type":"ContainerStarted","Data":"7415c9143b6652bf55e6d4fb4f23695302d6acc9e79498c1e1ce75829fbc82e7"} Sep 08 14:43:39.574523 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:39.574191 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:43:39.592537 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:39.592468 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" podStartSLOduration=1.937120852 podStartE2EDuration="2.592448084s" totalImagesPullingTime="655.327232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:43:39.591703213 +0000 UTC m=+1631.876020331" watchObservedRunningTime="2026-09-08 14:43:39.592448084 +0000 UTC m=+1631.876765200" Sep 08 14:43:45.585448 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:43:45.585418 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:44:08.106606 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.106568 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj"] Sep 08 14:44:08.107137 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.106860 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" podUID="14af086a-60d8-4e7e-b501-eb17f2de538d" containerName="maas-api" containerID="cri-o://7415c9143b6652bf55e6d4fb4f23695302d6acc9e79498c1e1ce75829fbc82e7" gracePeriod=30 Sep 08 14:44:08.362359 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.362303 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:44:08.391071 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.391039 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-maas-api-tls\") pod \"14af086a-60d8-4e7e-b501-eb17f2de538d\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " Sep 08 14:44:08.391202 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.391172 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-metrics-certs\") pod \"14af086a-60d8-4e7e-b501-eb17f2de538d\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " Sep 08 14:44:08.391254 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.391206 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14af086a-60d8-4e7e-b501-eb17f2de538d-openshift-service-ca\") pod \"14af086a-60d8-4e7e-b501-eb17f2de538d\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " Sep 08 14:44:08.391254 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.391242 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rtl6g\" (UniqueName: \"kubernetes.io/projected/14af086a-60d8-4e7e-b501-eb17f2de538d-kube-api-access-rtl6g\") pod \"14af086a-60d8-4e7e-b501-eb17f2de538d\" (UID: \"14af086a-60d8-4e7e-b501-eb17f2de538d\") " Sep 08 14:44:08.391612 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.391584 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14af086a-60d8-4e7e-b501-eb17f2de538d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "14af086a-60d8-4e7e-b501-eb17f2de538d" (UID: "14af086a-60d8-4e7e-b501-eb17f2de538d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:44:08.393274 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.393250 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "14af086a-60d8-4e7e-b501-eb17f2de538d" (UID: "14af086a-60d8-4e7e-b501-eb17f2de538d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:44:08.393366 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.393345 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14af086a-60d8-4e7e-b501-eb17f2de538d-kube-api-access-rtl6g" (OuterVolumeSpecName: "kube-api-access-rtl6g") pod "14af086a-60d8-4e7e-b501-eb17f2de538d" (UID: "14af086a-60d8-4e7e-b501-eb17f2de538d"). InnerVolumeSpecName "kube-api-access-rtl6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:44:08.393424 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.393407 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "14af086a-60d8-4e7e-b501-eb17f2de538d" (UID: "14af086a-60d8-4e7e-b501-eb17f2de538d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:44:08.492948 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.492918 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtl6g\" (UniqueName: \"kubernetes.io/projected/14af086a-60d8-4e7e-b501-eb17f2de538d-kube-api-access-rtl6g\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:44:08.492948 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.492943 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:44:08.492948 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.492952 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14af086a-60d8-4e7e-b501-eb17f2de538d-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:44:08.493168 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.492960 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14af086a-60d8-4e7e-b501-eb17f2de538d-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:44:08.740056 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.740022 2578 generic.go:356] "Generic (PLEG): container finished" podID="14af086a-60d8-4e7e-b501-eb17f2de538d" containerID="7415c9143b6652bf55e6d4fb4f23695302d6acc9e79498c1e1ce75829fbc82e7" exitCode=0 Sep 08 14:44:08.740235 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.740069 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" Sep 08 14:44:08.740235 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.740108 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" event={"ID":"14af086a-60d8-4e7e-b501-eb17f2de538d","Type":"ContainerDied","Data":"7415c9143b6652bf55e6d4fb4f23695302d6acc9e79498c1e1ce75829fbc82e7"} Sep 08 14:44:08.740235 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.740167 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj" event={"ID":"14af086a-60d8-4e7e-b501-eb17f2de538d","Type":"ContainerDied","Data":"c9e6dcbd9912bfd15450ccf7ca07546e6153dc3f282324d2b1d4d276493b6d2b"} Sep 08 14:44:08.740235 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.740194 2578 scope.go:117] "RemoveContainer" containerID="7415c9143b6652bf55e6d4fb4f23695302d6acc9e79498c1e1ce75829fbc82e7" Sep 08 14:44:08.750517 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.750500 2578 scope.go:117] "RemoveContainer" containerID="7415c9143b6652bf55e6d4fb4f23695302d6acc9e79498c1e1ce75829fbc82e7" Sep 08 14:44:08.750797 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:44:08.750777 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7415c9143b6652bf55e6d4fb4f23695302d6acc9e79498c1e1ce75829fbc82e7\": container with ID starting with 7415c9143b6652bf55e6d4fb4f23695302d6acc9e79498c1e1ce75829fbc82e7 not found: ID does not exist" containerID="7415c9143b6652bf55e6d4fb4f23695302d6acc9e79498c1e1ce75829fbc82e7" Sep 08 14:44:08.750847 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.750804 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7415c9143b6652bf55e6d4fb4f23695302d6acc9e79498c1e1ce75829fbc82e7"} err="failed to get container status \"7415c9143b6652bf55e6d4fb4f23695302d6acc9e79498c1e1ce75829fbc82e7\": rpc error: code = NotFound desc = could not find container \"7415c9143b6652bf55e6d4fb4f23695302d6acc9e79498c1e1ce75829fbc82e7\": container with ID starting with 7415c9143b6652bf55e6d4fb4f23695302d6acc9e79498c1e1ce75829fbc82e7 not found: ID does not exist" Sep 08 14:44:08.765410 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.765387 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj"] Sep 08 14:44:08.767781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:08.767762 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-6b7995-5d5cb4fb79-fxvqj"] Sep 08 14:44:10.241659 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:10.241625 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14af086a-60d8-4e7e-b501-eb17f2de538d" path="/var/lib/kubelet/pods/14af086a-60d8-4e7e-b501-eb17f2de538d/volumes" Sep 08 14:44:34.653167 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:34.653133 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l"] Sep 08 14:44:34.653767 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:34.653344 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" podUID="9813e7de-1f89-4b70-bff6-891edd0704e3" containerName="maas-api" containerID="cri-o://72544ade6d098e55cd8e309962ae08d790457d76d01c550ff0ab2d4665cf6069" gracePeriod=30 Sep 08 14:44:34.889499 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:34.889474 2578 generic.go:356] "Generic (PLEG): container finished" podID="9813e7de-1f89-4b70-bff6-891edd0704e3" containerID="72544ade6d098e55cd8e309962ae08d790457d76d01c550ff0ab2d4665cf6069" exitCode=0 Sep 08 14:44:34.889632 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:34.889548 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" event={"ID":"9813e7de-1f89-4b70-bff6-891edd0704e3","Type":"ContainerDied","Data":"72544ade6d098e55cd8e309962ae08d790457d76d01c550ff0ab2d4665cf6069"} Sep 08 14:44:34.889632 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:34.889611 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" event={"ID":"9813e7de-1f89-4b70-bff6-891edd0704e3","Type":"ContainerDied","Data":"f8bbca793aa22e64213f28678042aeff5149fefa8011c24d70d90dd03eda7d75"} Sep 08 14:44:34.889721 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:34.889632 2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8bbca793aa22e64213f28678042aeff5149fefa8011c24d70d90dd03eda7d75" Sep 08 14:44:34.904268 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:34.904212 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:44:35.043724 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.043697 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-maas-api-tls\") pod \"9813e7de-1f89-4b70-bff6-891edd0704e3\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " Sep 08 14:44:35.043724 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.043752 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-metrics-certs\") pod \"9813e7de-1f89-4b70-bff6-891edd0704e3\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " Sep 08 14:44:35.044071 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.043917 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdss5\" (UniqueName: \"kubernetes.io/projected/9813e7de-1f89-4b70-bff6-891edd0704e3-kube-api-access-fdss5\") pod \"9813e7de-1f89-4b70-bff6-891edd0704e3\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " Sep 08 14:44:35.044071 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.044005 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9813e7de-1f89-4b70-bff6-891edd0704e3-openshift-service-ca\") pod \"9813e7de-1f89-4b70-bff6-891edd0704e3\" (UID: \"9813e7de-1f89-4b70-bff6-891edd0704e3\") " Sep 08 14:44:35.044833 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.044377 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9813e7de-1f89-4b70-bff6-891edd0704e3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9813e7de-1f89-4b70-bff6-891edd0704e3" (UID: "9813e7de-1f89-4b70-bff6-891edd0704e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:44:35.044833 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.044653 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9813e7de-1f89-4b70-bff6-891edd0704e3-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:44:35.046047 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.046024 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9813e7de-1f89-4b70-bff6-891edd0704e3" (UID: "9813e7de-1f89-4b70-bff6-891edd0704e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:44:35.046161 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.046047 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9813e7de-1f89-4b70-bff6-891edd0704e3" (UID: "9813e7de-1f89-4b70-bff6-891edd0704e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:44:35.046261 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.046245 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9813e7de-1f89-4b70-bff6-891edd0704e3-kube-api-access-fdss5" (OuterVolumeSpecName: "kube-api-access-fdss5") pod "9813e7de-1f89-4b70-bff6-891edd0704e3" (UID: "9813e7de-1f89-4b70-bff6-891edd0704e3"). InnerVolumeSpecName "kube-api-access-fdss5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:44:35.146096 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.146066 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:44:35.146096 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.146090 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9813e7de-1f89-4b70-bff6-891edd0704e3-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:44:35.146096 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.146098 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdss5\" (UniqueName: \"kubernetes.io/projected/9813e7de-1f89-4b70-bff6-891edd0704e3-kube-api-access-fdss5\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:44:35.895332 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.895301 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l" Sep 08 14:44:35.918242 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.918218 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l"] Sep 08 14:44:35.922539 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:35.922517 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-274ffa-84dd86b5fd-jg75l"] Sep 08 14:44:36.241674 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:44:36.241640 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9813e7de-1f89-4b70-bff6-891edd0704e3" path="/var/lib/kubelet/pods/9813e7de-1f89-4b70-bff6-891edd0704e3/volumes" Sep 08 14:45:35.253005 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.252972 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288"] Sep 08 14:45:35.253587 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.253185 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" podUID="7ab8c321-5664-4047-824b-00bfc7501c47" containerName="maas-api" containerID="cri-o://a112e201e940d109a0700e33464b1eee365daffb98795d78ca0ea0cbe9e5adbd" gracePeriod=30 Sep 08 14:45:35.510153 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.510073 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:45:35.608477 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.608447 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ws2ms\" (UniqueName: \"kubernetes.io/projected/7ab8c321-5664-4047-824b-00bfc7501c47-kube-api-access-ws2ms\") pod \"7ab8c321-5664-4047-824b-00bfc7501c47\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " Sep 08 14:45:35.608648 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.608539 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab8c321-5664-4047-824b-00bfc7501c47-maas-api-tls\") pod \"7ab8c321-5664-4047-824b-00bfc7501c47\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " Sep 08 14:45:35.608648 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.608594 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab8c321-5664-4047-824b-00bfc7501c47-metrics-certs\") pod \"7ab8c321-5664-4047-824b-00bfc7501c47\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " Sep 08 14:45:35.608750 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.608665 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8c321-5664-4047-824b-00bfc7501c47-openshift-service-ca\") pod \"7ab8c321-5664-4047-824b-00bfc7501c47\" (UID: \"7ab8c321-5664-4047-824b-00bfc7501c47\") " Sep 08 14:45:35.609096 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.609062 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab8c321-5664-4047-824b-00bfc7501c47-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ab8c321-5664-4047-824b-00bfc7501c47" (UID: "7ab8c321-5664-4047-824b-00bfc7501c47"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:45:35.610872 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.610849 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab8c321-5664-4047-824b-00bfc7501c47-kube-api-access-ws2ms" (OuterVolumeSpecName: "kube-api-access-ws2ms") pod "7ab8c321-5664-4047-824b-00bfc7501c47" (UID: "7ab8c321-5664-4047-824b-00bfc7501c47"). InnerVolumeSpecName "kube-api-access-ws2ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:45:35.610998 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.610889 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab8c321-5664-4047-824b-00bfc7501c47-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ab8c321-5664-4047-824b-00bfc7501c47" (UID: "7ab8c321-5664-4047-824b-00bfc7501c47"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:45:35.610998 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.610908 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab8c321-5664-4047-824b-00bfc7501c47-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ab8c321-5664-4047-824b-00bfc7501c47" (UID: "7ab8c321-5664-4047-824b-00bfc7501c47"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:45:35.709965 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.709929 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ab8c321-5664-4047-824b-00bfc7501c47-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:45:35.709965 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.709958 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab8c321-5664-4047-824b-00bfc7501c47-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:45:35.709965 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.709967 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ab8c321-5664-4047-824b-00bfc7501c47-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:45:35.710210 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:35.709984 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ws2ms\" (UniqueName: \"kubernetes.io/projected/7ab8c321-5664-4047-824b-00bfc7501c47-kube-api-access-ws2ms\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:45:36.252214 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:36.252182 2578 generic.go:356] "Generic (PLEG): container finished" podID="7ab8c321-5664-4047-824b-00bfc7501c47" containerID="a112e201e940d109a0700e33464b1eee365daffb98795d78ca0ea0cbe9e5adbd" exitCode=0 Sep 08 14:45:36.252382 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:36.252226 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" Sep 08 14:45:36.252382 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:36.252262 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" event={"ID":"7ab8c321-5664-4047-824b-00bfc7501c47","Type":"ContainerDied","Data":"a112e201e940d109a0700e33464b1eee365daffb98795d78ca0ea0cbe9e5adbd"} Sep 08 14:45:36.252382 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:36.252326 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288" event={"ID":"7ab8c321-5664-4047-824b-00bfc7501c47","Type":"ContainerDied","Data":"fc8b3d19ff1b66a4efe25c7110c4ca51e818b2b420db8a4aab50a6d875481a5b"} Sep 08 14:45:36.252382 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:36.252356 2578 scope.go:117] "RemoveContainer" containerID="a112e201e940d109a0700e33464b1eee365daffb98795d78ca0ea0cbe9e5adbd" Sep 08 14:45:36.262667 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:36.262513 2578 scope.go:117] "RemoveContainer" containerID="a112e201e940d109a0700e33464b1eee365daffb98795d78ca0ea0cbe9e5adbd" Sep 08 14:45:36.262921 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:45:36.262837 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a112e201e940d109a0700e33464b1eee365daffb98795d78ca0ea0cbe9e5adbd\": container with ID starting with a112e201e940d109a0700e33464b1eee365daffb98795d78ca0ea0cbe9e5adbd not found: ID does not exist" containerID="a112e201e940d109a0700e33464b1eee365daffb98795d78ca0ea0cbe9e5adbd" Sep 08 14:45:36.262921 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:36.262863 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a112e201e940d109a0700e33464b1eee365daffb98795d78ca0ea0cbe9e5adbd"} err="failed to get container status \"a112e201e940d109a0700e33464b1eee365daffb98795d78ca0ea0cbe9e5adbd\": rpc error: code = NotFound desc = could not find container \"a112e201e940d109a0700e33464b1eee365daffb98795d78ca0ea0cbe9e5adbd\": container with ID starting with a112e201e940d109a0700e33464b1eee365daffb98795d78ca0ea0cbe9e5adbd not found: ID does not exist" Sep 08 14:45:36.269560 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:36.269538 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288"] Sep 08 14:45:36.275356 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:36.275331 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f7d00a-5f4968f47-84288"] Sep 08 14:45:38.241402 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:38.241371 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ab8c321-5664-4047-824b-00bfc7501c47" path="/var/lib/kubelet/pods/7ab8c321-5664-4047-824b-00bfc7501c47/volumes" Sep 08 14:45:53.795599 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:53.795558 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj"] Sep 08 14:45:53.796163 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:53.795843 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" podUID="b05126ef-a20b-4027-85a5-8f0ea79c7d8d" containerName="maas-api" containerID="cri-o://38e801b5e116e50b6d29efba1255eb86939be8cf04988a9060c1464740c6a185" gracePeriod=30 Sep 08 14:45:54.041140 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.041112 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:45:54.187911 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.187815 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-openshift-service-ca\") pod \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " Sep 08 14:45:54.187911 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.187875 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-maas-api-tls\") pod \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " Sep 08 14:45:54.188161 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.187945 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-metrics-certs\") pod \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " Sep 08 14:45:54.188161 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.187973 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t87zj\" (UniqueName: \"kubernetes.io/projected/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-kube-api-access-t87zj\") pod \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\" (UID: \"b05126ef-a20b-4027-85a5-8f0ea79c7d8d\") " Sep 08 14:45:54.188270 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.188207 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b05126ef-a20b-4027-85a5-8f0ea79c7d8d" (UID: "b05126ef-a20b-4027-85a5-8f0ea79c7d8d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:45:54.190315 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.190289 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b05126ef-a20b-4027-85a5-8f0ea79c7d8d" (UID: "b05126ef-a20b-4027-85a5-8f0ea79c7d8d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:45:54.190457 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.190432 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-kube-api-access-t87zj" (OuterVolumeSpecName: "kube-api-access-t87zj") pod "b05126ef-a20b-4027-85a5-8f0ea79c7d8d" (UID: "b05126ef-a20b-4027-85a5-8f0ea79c7d8d"). InnerVolumeSpecName "kube-api-access-t87zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:45:54.190587 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.190563 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b05126ef-a20b-4027-85a5-8f0ea79c7d8d" (UID: "b05126ef-a20b-4027-85a5-8f0ea79c7d8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:45:54.289705 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.289674 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:45:54.289705 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.289701 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:45:54.289705 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.289710 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t87zj\" (UniqueName: \"kubernetes.io/projected/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-kube-api-access-t87zj\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:45:54.289967 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.289721 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b05126ef-a20b-4027-85a5-8f0ea79c7d8d-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:45:54.365543 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.365512 2578 generic.go:356] "Generic (PLEG): container finished" podID="b05126ef-a20b-4027-85a5-8f0ea79c7d8d" containerID="38e801b5e116e50b6d29efba1255eb86939be8cf04988a9060c1464740c6a185" exitCode=0 Sep 08 14:45:54.365773 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.365578 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" Sep 08 14:45:54.365773 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.365609 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" event={"ID":"b05126ef-a20b-4027-85a5-8f0ea79c7d8d","Type":"ContainerDied","Data":"38e801b5e116e50b6d29efba1255eb86939be8cf04988a9060c1464740c6a185"} Sep 08 14:45:54.365773 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.365664 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj" event={"ID":"b05126ef-a20b-4027-85a5-8f0ea79c7d8d","Type":"ContainerDied","Data":"ab73b1a111b8f3819de8491cfb29db93d719fe62c9d24eed99c7bf4b3bcf0d65"} Sep 08 14:45:54.365773 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.365692 2578 scope.go:117] "RemoveContainer" containerID="38e801b5e116e50b6d29efba1255eb86939be8cf04988a9060c1464740c6a185" Sep 08 14:45:54.382794 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.382767 2578 scope.go:117] "RemoveContainer" containerID="38e801b5e116e50b6d29efba1255eb86939be8cf04988a9060c1464740c6a185" Sep 08 14:45:54.383338 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:45:54.383297 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38e801b5e116e50b6d29efba1255eb86939be8cf04988a9060c1464740c6a185\": container with ID starting with 38e801b5e116e50b6d29efba1255eb86939be8cf04988a9060c1464740c6a185 not found: ID does not exist" containerID="38e801b5e116e50b6d29efba1255eb86939be8cf04988a9060c1464740c6a185" Sep 08 14:45:54.383454 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.383382 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38e801b5e116e50b6d29efba1255eb86939be8cf04988a9060c1464740c6a185"} err="failed to get container status \"38e801b5e116e50b6d29efba1255eb86939be8cf04988a9060c1464740c6a185\": rpc error: code = NotFound desc = could not find container \"38e801b5e116e50b6d29efba1255eb86939be8cf04988a9060c1464740c6a185\": container with ID starting with 38e801b5e116e50b6d29efba1255eb86939be8cf04988a9060c1464740c6a185 not found: ID does not exist" Sep 08 14:45:54.386521 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.386499 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj"] Sep 08 14:45:54.391086 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:54.391067 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-780944-5d9c98b85f-9xjkj"] Sep 08 14:45:56.241839 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:45:56.241801 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b05126ef-a20b-4027-85a5-8f0ea79c7d8d" path="/var/lib/kubelet/pods/b05126ef-a20b-4027-85a5-8f0ea79c7d8d/volumes" Sep 08 14:46:12.350499 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.350466 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw"] Sep 08 14:46:12.350997 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.350704 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" podUID="4a5d4802-46e4-463a-8db9-01bd3f35768c" containerName="maas-api" containerID="cri-o://afeeb00d6b9439b71a9ef2b093d28a2e67a7e98ddd040dbb58c55c5906b0cdab" gracePeriod=30 Sep 08 14:46:12.472370 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.472341 2578 generic.go:356] "Generic (PLEG): container finished" podID="4a5d4802-46e4-463a-8db9-01bd3f35768c" containerID="afeeb00d6b9439b71a9ef2b093d28a2e67a7e98ddd040dbb58c55c5906b0cdab" exitCode=0 Sep 08 14:46:12.472510 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.472417 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" event={"ID":"4a5d4802-46e4-463a-8db9-01bd3f35768c","Type":"ContainerDied","Data":"afeeb00d6b9439b71a9ef2b093d28a2e67a7e98ddd040dbb58c55c5906b0cdab"} Sep 08 14:46:12.602537 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.602471 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:46:12.663494 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.663455 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5d4802-46e4-463a-8db9-01bd3f35768c-openshift-service-ca\") pod \"4a5d4802-46e4-463a-8db9-01bd3f35768c\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " Sep 08 14:46:12.663494 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.663500 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5d4802-46e4-463a-8db9-01bd3f35768c-metrics-certs\") pod \"4a5d4802-46e4-463a-8db9-01bd3f35768c\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " Sep 08 14:46:12.663779 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.663526 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5d4802-46e4-463a-8db9-01bd3f35768c-maas-api-tls\") pod \"4a5d4802-46e4-463a-8db9-01bd3f35768c\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " Sep 08 14:46:12.663779 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.663629 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9ddm\" (UniqueName: \"kubernetes.io/projected/4a5d4802-46e4-463a-8db9-01bd3f35768c-kube-api-access-w9ddm\") pod \"4a5d4802-46e4-463a-8db9-01bd3f35768c\" (UID: \"4a5d4802-46e4-463a-8db9-01bd3f35768c\") " Sep 08 14:46:12.663912 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.663882 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5d4802-46e4-463a-8db9-01bd3f35768c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a5d4802-46e4-463a-8db9-01bd3f35768c" (UID: "4a5d4802-46e4-463a-8db9-01bd3f35768c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:46:12.665870 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.665845 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5d4802-46e4-463a-8db9-01bd3f35768c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a5d4802-46e4-463a-8db9-01bd3f35768c" (UID: "4a5d4802-46e4-463a-8db9-01bd3f35768c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:46:12.665870 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.665857 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5d4802-46e4-463a-8db9-01bd3f35768c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4a5d4802-46e4-463a-8db9-01bd3f35768c" (UID: "4a5d4802-46e4-463a-8db9-01bd3f35768c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:46:12.665977 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.665845 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5d4802-46e4-463a-8db9-01bd3f35768c-kube-api-access-w9ddm" (OuterVolumeSpecName: "kube-api-access-w9ddm") pod "4a5d4802-46e4-463a-8db9-01bd3f35768c" (UID: "4a5d4802-46e4-463a-8db9-01bd3f35768c"). InnerVolumeSpecName "kube-api-access-w9ddm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:46:12.764657 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.764625 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9ddm\" (UniqueName: \"kubernetes.io/projected/4a5d4802-46e4-463a-8db9-01bd3f35768c-kube-api-access-w9ddm\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:46:12.764657 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.764651 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5d4802-46e4-463a-8db9-01bd3f35768c-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:46:12.764657 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.764661 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5d4802-46e4-463a-8db9-01bd3f35768c-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:46:12.764923 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:12.764669 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5d4802-46e4-463a-8db9-01bd3f35768c-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:46:13.478944 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:13.478912 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" Sep 08 14:46:13.478944 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:13.478940 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw" event={"ID":"4a5d4802-46e4-463a-8db9-01bd3f35768c","Type":"ContainerDied","Data":"d252fa19cdb1a28d3eebbb3c5f81170b76a4ff104c66c20d1ab7cdc09e9e34dd"} Sep 08 14:46:13.479407 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:13.478999 2578 scope.go:117] "RemoveContainer" containerID="afeeb00d6b9439b71a9ef2b093d28a2e67a7e98ddd040dbb58c55c5906b0cdab" Sep 08 14:46:13.501715 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:13.501687 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw"] Sep 08 14:46:13.508968 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:13.508931 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-017dbd-6dd7595cbb-hv2kw"] Sep 08 14:46:14.241884 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:14.241844 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a5d4802-46e4-463a-8db9-01bd3f35768c" path="/var/lib/kubelet/pods/4a5d4802-46e4-463a-8db9-01bd3f35768c/volumes" Sep 08 14:46:22.504688 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:22.504605 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb4b89648-78rbv"] Sep 08 14:46:22.505133 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:22.504830 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6cb4b89648-78rbv" podUID="e55c94b0-218e-4184-abd6-ad53f40b8b19" containerName="manager" containerID="cri-o://b2807e64c7584f1f52fe4a5006ec2bd5e98c8829774fd7454fd8eb1eb338b34b" gracePeriod=30 Sep 08 14:46:22.759598 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:22.759542 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:46:22.851058 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:22.851029 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e55c94b0-218e-4184-abd6-ad53f40b8b19-webhook-cert\") pod \"e55c94b0-218e-4184-abd6-ad53f40b8b19\" (UID: \"e55c94b0-218e-4184-abd6-ad53f40b8b19\") " Sep 08 14:46:22.851225 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:22.851115 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t962p\" (UniqueName: \"kubernetes.io/projected/e55c94b0-218e-4184-abd6-ad53f40b8b19-kube-api-access-t962p\") pod \"e55c94b0-218e-4184-abd6-ad53f40b8b19\" (UID: \"e55c94b0-218e-4184-abd6-ad53f40b8b19\") " Sep 08 14:46:22.851225 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:22.851132 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55c94b0-218e-4184-abd6-ad53f40b8b19-metrics-certs\") pod \"e55c94b0-218e-4184-abd6-ad53f40b8b19\" (UID: \"e55c94b0-218e-4184-abd6-ad53f40b8b19\") " Sep 08 14:46:22.853326 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:22.853294 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55c94b0-218e-4184-abd6-ad53f40b8b19-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e55c94b0-218e-4184-abd6-ad53f40b8b19" (UID: "e55c94b0-218e-4184-abd6-ad53f40b8b19"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:46:22.853438 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:22.853380 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e55c94b0-218e-4184-abd6-ad53f40b8b19-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e55c94b0-218e-4184-abd6-ad53f40b8b19" (UID: "e55c94b0-218e-4184-abd6-ad53f40b8b19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:46:22.853438 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:22.853386 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e55c94b0-218e-4184-abd6-ad53f40b8b19-kube-api-access-t962p" (OuterVolumeSpecName: "kube-api-access-t962p") pod "e55c94b0-218e-4184-abd6-ad53f40b8b19" (UID: "e55c94b0-218e-4184-abd6-ad53f40b8b19"). InnerVolumeSpecName "kube-api-access-t962p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:46:22.952606 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:22.952568 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t962p\" (UniqueName: \"kubernetes.io/projected/e55c94b0-218e-4184-abd6-ad53f40b8b19-kube-api-access-t962p\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:46:22.952606 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:22.952600 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e55c94b0-218e-4184-abd6-ad53f40b8b19-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:46:22.952606 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:22.952609 2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e55c94b0-218e-4184-abd6-ad53f40b8b19-webhook-cert\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:46:23.537436 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:23.537403 2578 generic.go:356] "Generic (PLEG): container finished" podID="e55c94b0-218e-4184-abd6-ad53f40b8b19" containerID="b2807e64c7584f1f52fe4a5006ec2bd5e98c8829774fd7454fd8eb1eb338b34b" exitCode=0 Sep 08 14:46:23.537868 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:23.537447 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb4b89648-78rbv" Sep 08 14:46:23.537868 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:23.537470 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb4b89648-78rbv" event={"ID":"e55c94b0-218e-4184-abd6-ad53f40b8b19","Type":"ContainerDied","Data":"b2807e64c7584f1f52fe4a5006ec2bd5e98c8829774fd7454fd8eb1eb338b34b"} Sep 08 14:46:23.537868 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:23.537510 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb4b89648-78rbv" event={"ID":"e55c94b0-218e-4184-abd6-ad53f40b8b19","Type":"ContainerDied","Data":"44e8cb12cb5ba3976dae5585b806740b25b70180450bbd223916180e5bd75a68"} Sep 08 14:46:23.537868 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:23.537530 2578 scope.go:117] "RemoveContainer" containerID="b2807e64c7584f1f52fe4a5006ec2bd5e98c8829774fd7454fd8eb1eb338b34b" Sep 08 14:46:23.547763 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:23.547720 2578 scope.go:117] "RemoveContainer" containerID="b2807e64c7584f1f52fe4a5006ec2bd5e98c8829774fd7454fd8eb1eb338b34b" Sep 08 14:46:23.548007 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:46:23.547986 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2807e64c7584f1f52fe4a5006ec2bd5e98c8829774fd7454fd8eb1eb338b34b\": container with ID starting with b2807e64c7584f1f52fe4a5006ec2bd5e98c8829774fd7454fd8eb1eb338b34b not found: ID does not exist" containerID="b2807e64c7584f1f52fe4a5006ec2bd5e98c8829774fd7454fd8eb1eb338b34b" Sep 08 14:46:23.548070 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:23.548016 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2807e64c7584f1f52fe4a5006ec2bd5e98c8829774fd7454fd8eb1eb338b34b"} err="failed to get container status \"b2807e64c7584f1f52fe4a5006ec2bd5e98c8829774fd7454fd8eb1eb338b34b\": rpc error: code = NotFound desc = could not find container \"b2807e64c7584f1f52fe4a5006ec2bd5e98c8829774fd7454fd8eb1eb338b34b\": container with ID starting with b2807e64c7584f1f52fe4a5006ec2bd5e98c8829774fd7454fd8eb1eb338b34b not found: ID does not exist" Sep 08 14:46:23.559872 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:23.559851 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb4b89648-78rbv"] Sep 08 14:46:23.564945 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:23.564924 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cb4b89648-78rbv"] Sep 08 14:46:24.241779 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:24.241722 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e55c94b0-218e-4184-abd6-ad53f40b8b19" path="/var/lib/kubelet/pods/e55c94b0-218e-4184-abd6-ad53f40b8b19/volumes" Sep 08 14:46:28.327767 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:28.327722 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:46:28.330280 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:28.330262 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:46:28.335631 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:28.335607 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:46:28.338063 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:46:28.338046 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:47:24.381125 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381089 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cb4b89648-82kj5"] Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381453 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b05126ef-a20b-4027-85a5-8f0ea79c7d8d" containerName="maas-api" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381467 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b05126ef-a20b-4027-85a5-8f0ea79c7d8d" containerName="maas-api" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381481 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ab8c321-5664-4047-824b-00bfc7501c47" containerName="maas-api" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381489 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ab8c321-5664-4047-824b-00bfc7501c47" containerName="maas-api" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381502 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9813e7de-1f89-4b70-bff6-891edd0704e3" containerName="maas-api" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381508 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9813e7de-1f89-4b70-bff6-891edd0704e3" containerName="maas-api" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381514 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a5d4802-46e4-463a-8db9-01bd3f35768c" containerName="maas-api" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381520 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a5d4802-46e4-463a-8db9-01bd3f35768c" containerName="maas-api" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381535 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14af086a-60d8-4e7e-b501-eb17f2de538d" containerName="maas-api" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381540 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="14af086a-60d8-4e7e-b501-eb17f2de538d" containerName="maas-api" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381546 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e55c94b0-218e-4184-abd6-ad53f40b8b19" containerName="manager" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381551 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e55c94b0-218e-4184-abd6-ad53f40b8b19" containerName="manager" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381603 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9813e7de-1f89-4b70-bff6-891edd0704e3" containerName="maas-api" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381611 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e55c94b0-218e-4184-abd6-ad53f40b8b19" containerName="manager" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381618 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ab8c321-5664-4047-824b-00bfc7501c47" containerName="maas-api" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381624 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a5d4802-46e4-463a-8db9-01bd3f35768c" containerName="maas-api" Sep 08 14:47:24.381621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381630 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b05126ef-a20b-4027-85a5-8f0ea79c7d8d" containerName="maas-api" Sep 08 14:47:24.382185 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.381635 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="14af086a-60d8-4e7e-b501-eb17f2de538d" containerName="maas-api" Sep 08 14:47:24.384829 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.384812 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:24.388074 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.388048 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-czc9x\"" Sep 08 14:47:24.388176 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.388082 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 14:47:24.388176 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.388102 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 14:47:24.388176 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.388082 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 14:47:24.394605 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.394574 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb4b89648-82kj5"] Sep 08 14:47:24.525237 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.525201 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzvtw\" (UniqueName: \"kubernetes.io/projected/b786df07-dd02-4ef0-9d75-dc70fb424e41-kube-api-access-tzvtw\") pod \"maas-controller-6cb4b89648-82kj5\" (UID: \"b786df07-dd02-4ef0-9d75-dc70fb424e41\") " pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:24.525404 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.525244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b786df07-dd02-4ef0-9d75-dc70fb424e41-webhook-cert\") pod \"maas-controller-6cb4b89648-82kj5\" (UID: \"b786df07-dd02-4ef0-9d75-dc70fb424e41\") " pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:24.525404 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.525336 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b786df07-dd02-4ef0-9d75-dc70fb424e41-metrics-certs\") pod \"maas-controller-6cb4b89648-82kj5\" (UID: \"b786df07-dd02-4ef0-9d75-dc70fb424e41\") " pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:24.626392 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.626358 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzvtw\" (UniqueName: \"kubernetes.io/projected/b786df07-dd02-4ef0-9d75-dc70fb424e41-kube-api-access-tzvtw\") pod \"maas-controller-6cb4b89648-82kj5\" (UID: \"b786df07-dd02-4ef0-9d75-dc70fb424e41\") " pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:24.626553 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.626398 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b786df07-dd02-4ef0-9d75-dc70fb424e41-webhook-cert\") pod \"maas-controller-6cb4b89648-82kj5\" (UID: \"b786df07-dd02-4ef0-9d75-dc70fb424e41\") " pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:24.626553 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.626456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b786df07-dd02-4ef0-9d75-dc70fb424e41-metrics-certs\") pod \"maas-controller-6cb4b89648-82kj5\" (UID: \"b786df07-dd02-4ef0-9d75-dc70fb424e41\") " pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:24.629117 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.629093 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b786df07-dd02-4ef0-9d75-dc70fb424e41-metrics-certs\") pod \"maas-controller-6cb4b89648-82kj5\" (UID: \"b786df07-dd02-4ef0-9d75-dc70fb424e41\") " pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:24.629224 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.629140 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b786df07-dd02-4ef0-9d75-dc70fb424e41-webhook-cert\") pod \"maas-controller-6cb4b89648-82kj5\" (UID: \"b786df07-dd02-4ef0-9d75-dc70fb424e41\") " pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:24.635029 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.634983 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzvtw\" (UniqueName: \"kubernetes.io/projected/b786df07-dd02-4ef0-9d75-dc70fb424e41-kube-api-access-tzvtw\") pod \"maas-controller-6cb4b89648-82kj5\" (UID: \"b786df07-dd02-4ef0-9d75-dc70fb424e41\") " pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:24.696358 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.696326 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:24.974877 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.974850 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb4b89648-82kj5"] Sep 08 14:47:24.975299 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:47:24.975267 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb786df07_dd02_4ef0_9d75_dc70fb424e41.slice/crio-66a7c9d04a8295f1492f4e5c12496f73ddfd5f052dd82501d6ae7966a90c063e WatchSource:0}: Error finding container 66a7c9d04a8295f1492f4e5c12496f73ddfd5f052dd82501d6ae7966a90c063e: Status 404 returned error can't find the container with id 66a7c9d04a8295f1492f4e5c12496f73ddfd5f052dd82501d6ae7966a90c063e Sep 08 14:47:24.976700 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:24.976685 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 08 14:47:25.896191 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:25.896159 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb4b89648-82kj5" event={"ID":"b786df07-dd02-4ef0-9d75-dc70fb424e41","Type":"ContainerStarted","Data":"c5f6d36aeb46debf2d4512b79360e05fd2a3bc16afd22d7e26ece416d2bb6813"} Sep 08 14:47:25.896582 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:25.896211 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb4b89648-82kj5" event={"ID":"b786df07-dd02-4ef0-9d75-dc70fb424e41","Type":"ContainerStarted","Data":"66a7c9d04a8295f1492f4e5c12496f73ddfd5f052dd82501d6ae7966a90c063e"} Sep 08 14:47:25.896582 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:25.896240 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:25.915442 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:25.915394 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cb4b89648-82kj5" podStartSLOduration=1.118747826 podStartE2EDuration="1.915377232s" totalImagesPullingTime="796.629406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:47:25.91272366 +0000 UTC m=+1858.197040775" watchObservedRunningTime="2026-09-08 14:47:25.915377232 +0000 UTC m=+1858.199694347" Sep 08 14:47:36.903959 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:36.903921 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:44.191909 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:44.191871 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb4b89648-82kj5"] Sep 08 14:47:44.192342 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:44.192162 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6cb4b89648-82kj5" podUID="b786df07-dd02-4ef0-9d75-dc70fb424e41" containerName="manager" containerID="cri-o://c5f6d36aeb46debf2d4512b79360e05fd2a3bc16afd22d7e26ece416d2bb6813" gracePeriod=30 Sep 08 14:47:44.457391 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:44.457370 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:44.505024 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:44.504987 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzvtw\" (UniqueName: \"kubernetes.io/projected/b786df07-dd02-4ef0-9d75-dc70fb424e41-kube-api-access-tzvtw\") pod \"b786df07-dd02-4ef0-9d75-dc70fb424e41\" (UID: \"b786df07-dd02-4ef0-9d75-dc70fb424e41\") " Sep 08 14:47:44.505234 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:44.505041 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b786df07-dd02-4ef0-9d75-dc70fb424e41-metrics-certs\") pod \"b786df07-dd02-4ef0-9d75-dc70fb424e41\" (UID: \"b786df07-dd02-4ef0-9d75-dc70fb424e41\") " Sep 08 14:47:44.505234 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:44.505123 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b786df07-dd02-4ef0-9d75-dc70fb424e41-webhook-cert\") pod \"b786df07-dd02-4ef0-9d75-dc70fb424e41\" (UID: \"b786df07-dd02-4ef0-9d75-dc70fb424e41\") " Sep 08 14:47:44.507352 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:44.507316 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b786df07-dd02-4ef0-9d75-dc70fb424e41-kube-api-access-tzvtw" (OuterVolumeSpecName: "kube-api-access-tzvtw") pod "b786df07-dd02-4ef0-9d75-dc70fb424e41" (UID: "b786df07-dd02-4ef0-9d75-dc70fb424e41"). InnerVolumeSpecName "kube-api-access-tzvtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:47:44.507856 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:44.507828 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b786df07-dd02-4ef0-9d75-dc70fb424e41-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b786df07-dd02-4ef0-9d75-dc70fb424e41" (UID: "b786df07-dd02-4ef0-9d75-dc70fb424e41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:47:44.507936 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:44.507852 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b786df07-dd02-4ef0-9d75-dc70fb424e41-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b786df07-dd02-4ef0-9d75-dc70fb424e41" (UID: "b786df07-dd02-4ef0-9d75-dc70fb424e41"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:47:44.606686 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:44.606654 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b786df07-dd02-4ef0-9d75-dc70fb424e41-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:47:44.606686 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:44.606681 2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b786df07-dd02-4ef0-9d75-dc70fb424e41-webhook-cert\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:47:44.606686 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:44.606690 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzvtw\" (UniqueName: \"kubernetes.io/projected/b786df07-dd02-4ef0-9d75-dc70fb424e41-kube-api-access-tzvtw\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:47:45.004569 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:45.004529 2578 generic.go:356] "Generic (PLEG): container finished" podID="b786df07-dd02-4ef0-9d75-dc70fb424e41" containerID="c5f6d36aeb46debf2d4512b79360e05fd2a3bc16afd22d7e26ece416d2bb6813" exitCode=0 Sep 08 14:47:45.004759 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:45.004580 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb4b89648-82kj5" Sep 08 14:47:45.004759 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:45.004613 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb4b89648-82kj5" event={"ID":"b786df07-dd02-4ef0-9d75-dc70fb424e41","Type":"ContainerDied","Data":"c5f6d36aeb46debf2d4512b79360e05fd2a3bc16afd22d7e26ece416d2bb6813"} Sep 08 14:47:45.004759 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:45.004691 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb4b89648-82kj5" event={"ID":"b786df07-dd02-4ef0-9d75-dc70fb424e41","Type":"ContainerDied","Data":"66a7c9d04a8295f1492f4e5c12496f73ddfd5f052dd82501d6ae7966a90c063e"} Sep 08 14:47:45.004759 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:45.004717 2578 scope.go:117] "RemoveContainer" containerID="c5f6d36aeb46debf2d4512b79360e05fd2a3bc16afd22d7e26ece416d2bb6813" Sep 08 14:47:45.015108 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:45.015089 2578 scope.go:117] "RemoveContainer" containerID="c5f6d36aeb46debf2d4512b79360e05fd2a3bc16afd22d7e26ece416d2bb6813" Sep 08 14:47:45.015394 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:47:45.015373 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5f6d36aeb46debf2d4512b79360e05fd2a3bc16afd22d7e26ece416d2bb6813\": container with ID starting with c5f6d36aeb46debf2d4512b79360e05fd2a3bc16afd22d7e26ece416d2bb6813 not found: ID does not exist" containerID="c5f6d36aeb46debf2d4512b79360e05fd2a3bc16afd22d7e26ece416d2bb6813" Sep 08 14:47:45.015458 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:45.015406 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5f6d36aeb46debf2d4512b79360e05fd2a3bc16afd22d7e26ece416d2bb6813"} err="failed to get container status \"c5f6d36aeb46debf2d4512b79360e05fd2a3bc16afd22d7e26ece416d2bb6813\": rpc error: code = NotFound desc = could not find container \"c5f6d36aeb46debf2d4512b79360e05fd2a3bc16afd22d7e26ece416d2bb6813\": container with ID starting with c5f6d36aeb46debf2d4512b79360e05fd2a3bc16afd22d7e26ece416d2bb6813 not found: ID does not exist" Sep 08 14:47:45.027861 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:45.027821 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb4b89648-82kj5"] Sep 08 14:47:45.032054 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:45.032031 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cb4b89648-82kj5"] Sep 08 14:47:46.129176 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.129145 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cb4b89648-l9xvs"] Sep 08 14:47:46.129547 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.129514 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b786df07-dd02-4ef0-9d75-dc70fb424e41" containerName="manager" Sep 08 14:47:46.129547 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.129525 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b786df07-dd02-4ef0-9d75-dc70fb424e41" containerName="manager" Sep 08 14:47:46.129619 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.129613 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b786df07-dd02-4ef0-9d75-dc70fb424e41" containerName="manager" Sep 08 14:47:46.133832 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.133817 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb4b89648-l9xvs" Sep 08 14:47:46.136919 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.136894 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Sep 08 14:47:46.136919 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.136908 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Sep 08 14:47:46.137209 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.137191 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-czc9x\"" Sep 08 14:47:46.137257 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.137191 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Sep 08 14:47:46.141035 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.141013 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb4b89648-l9xvs"] Sep 08 14:47:46.221417 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.221380 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5a1daac-ea5b-4f55-81c3-e9eb4a44351b-webhook-cert\") pod \"maas-controller-6cb4b89648-l9xvs\" (UID: \"a5a1daac-ea5b-4f55-81c3-e9eb4a44351b\") " pod="opendatahub/maas-controller-6cb4b89648-l9xvs" Sep 08 14:47:46.221658 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.221444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbqf8\" (UniqueName: \"kubernetes.io/projected/a5a1daac-ea5b-4f55-81c3-e9eb4a44351b-kube-api-access-mbqf8\") pod \"maas-controller-6cb4b89648-l9xvs\" (UID: \"a5a1daac-ea5b-4f55-81c3-e9eb4a44351b\") " pod="opendatahub/maas-controller-6cb4b89648-l9xvs" Sep 08 14:47:46.221658 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.221524 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1daac-ea5b-4f55-81c3-e9eb4a44351b-metrics-certs\") pod \"maas-controller-6cb4b89648-l9xvs\" (UID: \"a5a1daac-ea5b-4f55-81c3-e9eb4a44351b\") " pod="opendatahub/maas-controller-6cb4b89648-l9xvs" Sep 08 14:47:46.241216 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.241184 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b786df07-dd02-4ef0-9d75-dc70fb424e41" path="/var/lib/kubelet/pods/b786df07-dd02-4ef0-9d75-dc70fb424e41/volumes" Sep 08 14:47:46.322911 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.322875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5a1daac-ea5b-4f55-81c3-e9eb4a44351b-webhook-cert\") pod \"maas-controller-6cb4b89648-l9xvs\" (UID: \"a5a1daac-ea5b-4f55-81c3-e9eb4a44351b\") " pod="opendatahub/maas-controller-6cb4b89648-l9xvs" Sep 08 14:47:46.323114 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.322942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbqf8\" (UniqueName: \"kubernetes.io/projected/a5a1daac-ea5b-4f55-81c3-e9eb4a44351b-kube-api-access-mbqf8\") pod \"maas-controller-6cb4b89648-l9xvs\" (UID: \"a5a1daac-ea5b-4f55-81c3-e9eb4a44351b\") " pod="opendatahub/maas-controller-6cb4b89648-l9xvs" Sep 08 14:47:46.323114 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.323003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1daac-ea5b-4f55-81c3-e9eb4a44351b-metrics-certs\") pod \"maas-controller-6cb4b89648-l9xvs\" (UID: \"a5a1daac-ea5b-4f55-81c3-e9eb4a44351b\") " pod="opendatahub/maas-controller-6cb4b89648-l9xvs" Sep 08 14:47:46.325510 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.325483 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a5a1daac-ea5b-4f55-81c3-e9eb4a44351b-webhook-cert\") pod \"maas-controller-6cb4b89648-l9xvs\" (UID: \"a5a1daac-ea5b-4f55-81c3-e9eb4a44351b\") " pod="opendatahub/maas-controller-6cb4b89648-l9xvs" Sep 08 14:47:46.325616 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.325539 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a5a1daac-ea5b-4f55-81c3-e9eb4a44351b-metrics-certs\") pod \"maas-controller-6cb4b89648-l9xvs\" (UID: \"a5a1daac-ea5b-4f55-81c3-e9eb4a44351b\") " pod="opendatahub/maas-controller-6cb4b89648-l9xvs" Sep 08 14:47:46.332035 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.332011 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbqf8\" (UniqueName: \"kubernetes.io/projected/a5a1daac-ea5b-4f55-81c3-e9eb4a44351b-kube-api-access-mbqf8\") pod \"maas-controller-6cb4b89648-l9xvs\" (UID: \"a5a1daac-ea5b-4f55-81c3-e9eb4a44351b\") " pod="opendatahub/maas-controller-6cb4b89648-l9xvs" Sep 08 14:47:46.446445 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.446361 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb4b89648-l9xvs" Sep 08 14:47:46.719699 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:46.719674 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb4b89648-l9xvs"] Sep 08 14:47:46.721388 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:47:46.721361 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a1daac_ea5b_4f55_81c3_e9eb4a44351b.slice/crio-5042ec2923f1fb87e159d6f0340adcffed0763efdba94abaac66f7919b3fc0d3 WatchSource:0}: Error finding container 5042ec2923f1fb87e159d6f0340adcffed0763efdba94abaac66f7919b3fc0d3: Status 404 returned error can't find the container with id 5042ec2923f1fb87e159d6f0340adcffed0763efdba94abaac66f7919b3fc0d3 Sep 08 14:47:47.020420 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:47.020336 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb4b89648-l9xvs" event={"ID":"a5a1daac-ea5b-4f55-81c3-e9eb4a44351b","Type":"ContainerStarted","Data":"5042ec2923f1fb87e159d6f0340adcffed0763efdba94abaac66f7919b3fc0d3"} Sep 08 14:47:48.034592 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:48.034549 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb4b89648-l9xvs" event={"ID":"a5a1daac-ea5b-4f55-81c3-e9eb4a44351b","Type":"ContainerStarted","Data":"1247020ffcfcf4bac0401ada01cbca981fe8b55c62ff6cdce5f09a2579711d7d"} Sep 08 14:47:48.035124 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:48.034648 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cb4b89648-l9xvs" Sep 08 14:47:48.051752 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:48.051679 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cb4b89648-l9xvs" podStartSLOduration=1.501646133 podStartE2EDuration="2.051664559s" totalImagesPullingTime="550.018426ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:47:48.050618244 +0000 UTC m=+1880.334935382" watchObservedRunningTime="2026-09-08 14:47:48.051664559 +0000 UTC m=+1880.335981675" Sep 08 14:47:59.044552 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:47:59.044476 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cb4b89648-l9xvs" Sep 08 14:48:28.551657 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:48:28.551625 2578 scope.go:117] "RemoveContainer" containerID="72544ade6d098e55cd8e309962ae08d790457d76d01c550ff0ab2d4665cf6069" Sep 08 14:51:28.368096 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:28.368067 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:51:28.371715 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:28.371695 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:51:28.375901 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:28.375881 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:51:28.379981 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:28.379963 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tlbbx_1a3ac79a-e3b0-4b49-aa08-d77e1d12e397/ovn-acl-logging/0.log" Sep 08 14:51:33.896608 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:33.896574 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2"] Sep 08 14:51:33.900196 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:33.900177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:33.914963 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:33.914937 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2"] Sep 08 14:51:33.986217 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:33.986185 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/170ee046-ccb3-40f0-95dc-17891ec2c37e-openshift-service-ca\") pod \"maas-api-c5bcdfbdd-7gmx2\" (UID: \"170ee046-ccb3-40f0-95dc-17891ec2c37e\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:33.986391 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:33.986244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/170ee046-ccb3-40f0-95dc-17891ec2c37e-maas-api-tls\") pod \"maas-api-c5bcdfbdd-7gmx2\" (UID: \"170ee046-ccb3-40f0-95dc-17891ec2c37e\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:33.986391 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:33.986355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/170ee046-ccb3-40f0-95dc-17891ec2c37e-metrics-certs\") pod \"maas-api-c5bcdfbdd-7gmx2\" (UID: \"170ee046-ccb3-40f0-95dc-17891ec2c37e\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:33.986391 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:33.986383 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8g7q\" (UniqueName: \"kubernetes.io/projected/170ee046-ccb3-40f0-95dc-17891ec2c37e-kube-api-access-b8g7q\") pod \"maas-api-c5bcdfbdd-7gmx2\" (UID: \"170ee046-ccb3-40f0-95dc-17891ec2c37e\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:34.087236 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:34.087188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/170ee046-ccb3-40f0-95dc-17891ec2c37e-metrics-certs\") pod \"maas-api-c5bcdfbdd-7gmx2\" (UID: \"170ee046-ccb3-40f0-95dc-17891ec2c37e\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:34.087437 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:34.087248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8g7q\" (UniqueName: \"kubernetes.io/projected/170ee046-ccb3-40f0-95dc-17891ec2c37e-kube-api-access-b8g7q\") pod \"maas-api-c5bcdfbdd-7gmx2\" (UID: \"170ee046-ccb3-40f0-95dc-17891ec2c37e\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:34.087437 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:34.087298 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/170ee046-ccb3-40f0-95dc-17891ec2c37e-openshift-service-ca\") pod \"maas-api-c5bcdfbdd-7gmx2\" (UID: \"170ee046-ccb3-40f0-95dc-17891ec2c37e\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:34.087437 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:34.087364 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/170ee046-ccb3-40f0-95dc-17891ec2c37e-maas-api-tls\") pod \"maas-api-c5bcdfbdd-7gmx2\" (UID: \"170ee046-ccb3-40f0-95dc-17891ec2c37e\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:34.088192 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:34.088160 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/170ee046-ccb3-40f0-95dc-17891ec2c37e-openshift-service-ca\") pod \"maas-api-c5bcdfbdd-7gmx2\" (UID: \"170ee046-ccb3-40f0-95dc-17891ec2c37e\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:34.090089 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:34.090060 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/170ee046-ccb3-40f0-95dc-17891ec2c37e-metrics-certs\") pod \"maas-api-c5bcdfbdd-7gmx2\" (UID: \"170ee046-ccb3-40f0-95dc-17891ec2c37e\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:34.090442 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:34.090419 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/170ee046-ccb3-40f0-95dc-17891ec2c37e-maas-api-tls\") pod \"maas-api-c5bcdfbdd-7gmx2\" (UID: \"170ee046-ccb3-40f0-95dc-17891ec2c37e\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:34.096621 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:34.096592 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8g7q\" (UniqueName: \"kubernetes.io/projected/170ee046-ccb3-40f0-95dc-17891ec2c37e-kube-api-access-b8g7q\") pod \"maas-api-c5bcdfbdd-7gmx2\" (UID: \"170ee046-ccb3-40f0-95dc-17891ec2c37e\") " pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:34.212354 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:34.212323 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:34.498414 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:34.498385 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2"] Sep 08 14:51:34.501875 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:51:34.501843 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170ee046_ccb3_40f0_95dc_17891ec2c37e.slice/crio-14b68e6e2421f27a88e928683fb4e705144183f3912bc426f8cc1103bdd61808 WatchSource:0}: Error finding container 14b68e6e2421f27a88e928683fb4e705144183f3912bc426f8cc1103bdd61808: Status 404 returned error can't find the container with id 14b68e6e2421f27a88e928683fb4e705144183f3912bc426f8cc1103bdd61808 Sep 08 14:51:35.308979 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:35.308947 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" event={"ID":"170ee046-ccb3-40f0-95dc-17891ec2c37e","Type":"ContainerStarted","Data":"6e9c6a611f7935190b1939794bf804bbea51cf5fef7f6693cff01964fb911ae7"} Sep 08 14:51:35.309393 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:35.308995 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" event={"ID":"170ee046-ccb3-40f0-95dc-17891ec2c37e","Type":"ContainerStarted","Data":"14b68e6e2421f27a88e928683fb4e705144183f3912bc426f8cc1103bdd61808"} Sep 08 14:51:35.309393 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:35.309083 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:35.326928 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:35.326878 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" podStartSLOduration=1.889921405 podStartE2EDuration="2.326861845s" totalImagesPullingTime="436.94044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:51:35.32546378 +0000 UTC m=+2107.609780897" watchObservedRunningTime="2026-09-08 14:51:35.326861845 +0000 UTC m=+2107.611178959" Sep 08 14:51:41.320236 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:41.320206 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-c5bcdfbdd-7gmx2" Sep 08 14:51:48.652062 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:48.652028 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8554757b-b26bb"] Sep 08 14:51:48.652469 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:48.652237 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" podUID="1cd4800d-b7f7-43bc-a7de-f10389bd4ce9" containerName="maas-api" containerID="cri-o://b650880cab08637054e3afe0514c9a4d66032b316516325065abf84e2af0e959" gracePeriod=30 Sep 08 14:51:49.905206 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:49.905180 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:51:49.941448 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:49.941416 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-maas-api-tls\") pod \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " Sep 08 14:51:49.941618 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:49.941498 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-openshift-service-ca\") pod \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " Sep 08 14:51:49.941618 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:49.941550 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-metrics-certs\") pod \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " Sep 08 14:51:49.941618 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:49.941614 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrrbf\" (UniqueName: \"kubernetes.io/projected/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-kube-api-access-lrrbf\") pod \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\" (UID: \"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9\") " Sep 08 14:51:49.941918 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:49.941894 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1cd4800d-b7f7-43bc-a7de-f10389bd4ce9" (UID: "1cd4800d-b7f7-43bc-a7de-f10389bd4ce9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 08 14:51:49.943833 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:49.943800 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1cd4800d-b7f7-43bc-a7de-f10389bd4ce9" (UID: "1cd4800d-b7f7-43bc-a7de-f10389bd4ce9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:51:49.943947 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:49.943852 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1cd4800d-b7f7-43bc-a7de-f10389bd4ce9" (UID: "1cd4800d-b7f7-43bc-a7de-f10389bd4ce9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 08 14:51:49.943947 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:49.943853 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-kube-api-access-lrrbf" (OuterVolumeSpecName: "kube-api-access-lrrbf") pod "1cd4800d-b7f7-43bc-a7de-f10389bd4ce9" (UID: "1cd4800d-b7f7-43bc-a7de-f10389bd4ce9"). InnerVolumeSpecName "kube-api-access-lrrbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 08 14:51:50.042781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:50.042692 2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-openshift-service-ca\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:51:50.042781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:50.042717 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-metrics-certs\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:51:50.042781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:50.042751 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrrbf\" (UniqueName: \"kubernetes.io/projected/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-kube-api-access-lrrbf\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:51:50.042781 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:50.042760 2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9-maas-api-tls\") on node \"ip-10-0-141-155.ec2.internal\" DevicePath \"\"" Sep 08 14:51:50.392366 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:50.392273 2578 generic.go:356] "Generic (PLEG): container finished" podID="1cd4800d-b7f7-43bc-a7de-f10389bd4ce9" containerID="b650880cab08637054e3afe0514c9a4d66032b316516325065abf84e2af0e959" exitCode=0 Sep 08 14:51:50.392366 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:50.392321 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" Sep 08 14:51:50.392556 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:50.392364 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" event={"ID":"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9","Type":"ContainerDied","Data":"b650880cab08637054e3afe0514c9a4d66032b316516325065abf84e2af0e959"} Sep 08 14:51:50.392556 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:50.392419 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f8554757b-b26bb" event={"ID":"1cd4800d-b7f7-43bc-a7de-f10389bd4ce9","Type":"ContainerDied","Data":"a513072b18806808a538ccd8181c06a5106f7bec596db26c12a37a8d152f4d62"} Sep 08 14:51:50.392556 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:50.392448 2578 scope.go:117] "RemoveContainer" containerID="b650880cab08637054e3afe0514c9a4d66032b316516325065abf84e2af0e959" Sep 08 14:51:50.402003 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:50.401984 2578 scope.go:117] "RemoveContainer" containerID="b650880cab08637054e3afe0514c9a4d66032b316516325065abf84e2af0e959" Sep 08 14:51:50.402273 ip-10-0-141-155 kubenswrapper[2578]: E0908 14:51:50.402256 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b650880cab08637054e3afe0514c9a4d66032b316516325065abf84e2af0e959\": container with ID starting with b650880cab08637054e3afe0514c9a4d66032b316516325065abf84e2af0e959 not found: ID does not exist" containerID="b650880cab08637054e3afe0514c9a4d66032b316516325065abf84e2af0e959" Sep 08 14:51:50.402324 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:50.402282 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b650880cab08637054e3afe0514c9a4d66032b316516325065abf84e2af0e959"} err="failed to get container status \"b650880cab08637054e3afe0514c9a4d66032b316516325065abf84e2af0e959\": rpc error: code = NotFound desc = could not find container \"b650880cab08637054e3afe0514c9a4d66032b316516325065abf84e2af0e959\": container with ID starting with b650880cab08637054e3afe0514c9a4d66032b316516325065abf84e2af0e959 not found: ID does not exist" Sep 08 14:51:50.410821 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:50.410798 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8554757b-b26bb"] Sep 08 14:51:50.416758 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:50.416721 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f8554757b-b26bb"] Sep 08 14:51:52.242018 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:51:52.241983 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cd4800d-b7f7-43bc-a7de-f10389bd4ce9" path="/var/lib/kubelet/pods/1cd4800d-b7f7-43bc-a7de-f10389bd4ce9/volumes" Sep 08 14:52:07.146507 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:07.146472 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68d48c8894-jrdrc_ab33ab3a-7f69-4a89-9e69-6923da90e606/authorino/0.log" Sep 08 14:52:13.058223 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:13.058186 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cb4b89648-l9xvs_a5a1daac-ea5b-4f55-81c3-e9eb4a44351b/manager/0.log" Sep 08 14:52:13.434865 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:13.434782 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-566d57d45b-gwwdv_7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9/copy-manifests/0.log" Sep 08 14:52:13.442038 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:13.442014 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-566d57d45b-gwwdv_7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9/manager/0.log" Sep 08 14:52:14.175756 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:14.175708 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-c5bcdfbdd-7gmx2_170ee046-ccb3-40f0-95dc-17891ec2c37e/maas-api/0.log" Sep 08 14:52:14.532663 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:14.532635 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-2l4fk_85ae4a8f-59ef-4909-820c-20c177c7513f/postgres/0.log" Sep 08 14:52:15.370013 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:15.369980 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68d48c8894-jrdrc_ab33ab3a-7f69-4a89-9e69-6923da90e606/authorino/0.log" Sep 08 14:52:15.512558 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:15.512521 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-wplpz_83d27644-90c8-4fc3-a559-05754424ed05/manager/0.log" Sep 08 14:52:15.635564 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:15.635490 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-jpgwr_eceae5fa-0caf-43ce-a5c0-c9b65c745be3/manager/0.log" Sep 08 14:52:17.075509 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:17.075483 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jwqsz_07ac3b36-1d40-4ebe-a52f-31237baa182e/istio-proxy/1.log" Sep 08 14:52:17.199556 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:17.199531 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-kzbbd_a5385bcf-56ea-4679-87b6-44a041f2a606/payload-pre-processing/0.log" Sep 08 14:52:17.326699 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:17.326615 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9xh8f_88bb47cd-a680-4da8-8e34-092fb40d590d/payload-processing/0.log" Sep 08 14:52:18.086787 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:18.086757 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-7f626_112c52fa-a621-4eca-adda-3aa16e25a4d2/main/0.log" Sep 08 14:52:24.144972 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.144934 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rt5bz/must-gather-5g7bk"] Sep 08 14:52:24.145343 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.145316 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cd4800d-b7f7-43bc-a7de-f10389bd4ce9" containerName="maas-api" Sep 08 14:52:24.145343 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.145327 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cd4800d-b7f7-43bc-a7de-f10389bd4ce9" containerName="maas-api" Sep 08 14:52:24.145418 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.145398 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cd4800d-b7f7-43bc-a7de-f10389bd4ce9" containerName="maas-api" Sep 08 14:52:24.155107 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.155084 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rt5bz/must-gather-5g7bk" Sep 08 14:52:24.157447 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.157413 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rt5bz\"/\"default-dockercfg-fn8ct\"" Sep 08 14:52:24.157604 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.157492 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rt5bz\"/\"kube-root-ca.crt\"" Sep 08 14:52:24.158156 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.158136 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rt5bz\"/\"openshift-service-ca.crt\"" Sep 08 14:52:24.158946 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.158924 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rt5bz/must-gather-5g7bk"] Sep 08 14:52:24.256025 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.255997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/420bacc6-9af1-4a41-9dd7-e15611a4b96f-must-gather-output\") pod \"must-gather-5g7bk\" (UID: \"420bacc6-9af1-4a41-9dd7-e15611a4b96f\") " pod="openshift-must-gather-rt5bz/must-gather-5g7bk" Sep 08 14:52:24.256205 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.256119 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6lrv\" (UniqueName: \"kubernetes.io/projected/420bacc6-9af1-4a41-9dd7-e15611a4b96f-kube-api-access-n6lrv\") pod \"must-gather-5g7bk\" (UID: \"420bacc6-9af1-4a41-9dd7-e15611a4b96f\") " pod="openshift-must-gather-rt5bz/must-gather-5g7bk" Sep 08 14:52:24.356807 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.356761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6lrv\" (UniqueName: \"kubernetes.io/projected/420bacc6-9af1-4a41-9dd7-e15611a4b96f-kube-api-access-n6lrv\") pod \"must-gather-5g7bk\" (UID: \"420bacc6-9af1-4a41-9dd7-e15611a4b96f\") " pod="openshift-must-gather-rt5bz/must-gather-5g7bk" Sep 08 14:52:24.356952 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.356837 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/420bacc6-9af1-4a41-9dd7-e15611a4b96f-must-gather-output\") pod \"must-gather-5g7bk\" (UID: \"420bacc6-9af1-4a41-9dd7-e15611a4b96f\") " pod="openshift-must-gather-rt5bz/must-gather-5g7bk" Sep 08 14:52:24.357187 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.357168 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/420bacc6-9af1-4a41-9dd7-e15611a4b96f-must-gather-output\") pod \"must-gather-5g7bk\" (UID: \"420bacc6-9af1-4a41-9dd7-e15611a4b96f\") " pod="openshift-must-gather-rt5bz/must-gather-5g7bk" Sep 08 14:52:24.367814 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.367793 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6lrv\" (UniqueName: \"kubernetes.io/projected/420bacc6-9af1-4a41-9dd7-e15611a4b96f-kube-api-access-n6lrv\") pod \"must-gather-5g7bk\" (UID: \"420bacc6-9af1-4a41-9dd7-e15611a4b96f\") " pod="openshift-must-gather-rt5bz/must-gather-5g7bk" Sep 08 14:52:24.466319 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.466288 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rt5bz/must-gather-5g7bk" Sep 08 14:52:24.739526 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:24.739498 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rt5bz/must-gather-5g7bk"] Sep 08 14:52:24.740975 ip-10-0-141-155 kubenswrapper[2578]: W0908 14:52:24.740951 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod420bacc6_9af1_4a41_9dd7_e15611a4b96f.slice/crio-af776ad9e522e732d393653e5bc752397c2a80bbe26bbb9eb5d975f8d0053f0d WatchSource:0}: Error finding container af776ad9e522e732d393653e5bc752397c2a80bbe26bbb9eb5d975f8d0053f0d: Status 404 returned error can't find the container with id af776ad9e522e732d393653e5bc752397c2a80bbe26bbb9eb5d975f8d0053f0d Sep 08 14:52:25.592268 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:25.592226 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rt5bz/must-gather-5g7bk" event={"ID":"420bacc6-9af1-4a41-9dd7-e15611a4b96f","Type":"ContainerStarted","Data":"af776ad9e522e732d393653e5bc752397c2a80bbe26bbb9eb5d975f8d0053f0d"} Sep 08 14:52:28.613309 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:28.613267 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rt5bz/must-gather-5g7bk" event={"ID":"420bacc6-9af1-4a41-9dd7-e15611a4b96f","Type":"ContainerStarted","Data":"7d316bdd9af67e661fded59d11b7af2a343b7c95f3b4aa87f65730b8b325185f"} Sep 08 14:52:28.613798 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:28.613322 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rt5bz/must-gather-5g7bk" event={"ID":"420bacc6-9af1-4a41-9dd7-e15611a4b96f","Type":"ContainerStarted","Data":"e988bdb5cfa25e163849f39b19756def71753c227fc2d10be888805d83681cb2"} Sep 08 14:52:28.630849 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:28.630790 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rt5bz/must-gather-5g7bk" podStartSLOduration=1.316177559 podStartE2EDuration="4.630767318s" totalImagesPullingTime="3.314589759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:52:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:52:28.629466091 +0000 UTC m=+2160.913783205" watchObservedRunningTime="2026-09-08 14:52:28.630767318 +0000 UTC m=+2160.915084435" Sep 08 14:52:29.722111 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:29.722081 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hsh78_b4afcd3a-ef6b-4488-89b7-c48529821f6c/global-pull-secret-syncer/0.log" Sep 08 14:52:29.885771 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:29.885713 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tpxxk_2e80f39e-3303-4f76-b8aa-bccfff9edff6/konnectivity-agent/0.log" Sep 08 14:52:29.963565 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:29.963537 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-155.ec2.internal_308f44f9483500741181dba63fa91da4/haproxy/0.log" Sep 08 14:52:34.429154 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:34.429090 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68d48c8894-jrdrc_ab33ab3a-7f69-4a89-9e69-6923da90e606/authorino/0.log" Sep 08 14:52:34.470321 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:34.470289 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d88948d8-wplpz_83d27644-90c8-4fc3-a559-05754424ed05/manager/0.log" Sep 08 14:52:34.504583 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:34.503131 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-jpgwr_eceae5fa-0caf-43ce-a5c0-c9b65c745be3/manager/0.log" Sep 08 14:52:36.624168 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:36.624135 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-d9tcw_c5006d65-0045-4f1a-9946-f46325714d52/cluster-monitoring-operator/0.log" Sep 08 14:52:36.738375 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:36.738345 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-689794b998-v54kh_269e0a66-cd9b-442e-a6f3-7b83bca179cb/metrics-server/0.log" Sep 08 14:52:36.767761 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:36.767711 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-t56nk_6cf0f337-7106-4c18-9e39-2725ca0e8572/monitoring-plugin/0.log" Sep 08 14:52:36.885567 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:36.885486 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ptz62_e6b8a77b-7ec3-4c48-ae51-339d489290ed/node-exporter/0.log" Sep 08 14:52:36.909841 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:36.909809 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ptz62_e6b8a77b-7ec3-4c48-ae51-339d489290ed/kube-rbac-proxy/0.log" Sep 08 14:52:36.934602 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:36.934577 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ptz62_e6b8a77b-7ec3-4c48-ae51-339d489290ed/init-textfile/0.log" Sep 08 14:52:37.064720 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:37.064696 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xds8b_00557bb1-c1cd-469c-a493-ed035cf2b98b/kube-rbac-proxy-main/0.log" Sep 08 14:52:37.094687 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:37.094636 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xds8b_00557bb1-c1cd-469c-a493-ed035cf2b98b/kube-rbac-proxy-self/0.log" Sep 08 14:52:37.129807 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:37.129776 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-xds8b_00557bb1-c1cd-469c-a493-ed035cf2b98b/openshift-state-metrics/0.log" Sep 08 14:52:37.415569 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:37.415451 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-t7blt_b195df79-0181-4905-86b9-121fdbb67623/prometheus-operator/0.log" Sep 08 14:52:37.444067 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:37.444035 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-t7blt_b195df79-0181-4905-86b9-121fdbb67623/kube-rbac-proxy/0.log" Sep 08 14:52:37.473936 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:37.473912 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-zkqdx_a4b296eb-06db-4b56-8a44-8d4efea6e336/prometheus-operator-admission-webhook/0.log" Sep 08 14:52:37.512888 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:37.512861 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-98d849dd-mzr8j_ea434857-4c11-47cc-8b4e-ac465691ba3a/telemeter-client/0.log" Sep 08 14:52:37.536741 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:37.536696 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-98d849dd-mzr8j_ea434857-4c11-47cc-8b4e-ac465691ba3a/reload/0.log" Sep 08 14:52:37.563892 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:37.563866 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-98d849dd-mzr8j_ea434857-4c11-47cc-8b4e-ac465691ba3a/kube-rbac-proxy/0.log" Sep 08 14:52:39.217961 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:39.217928 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-2x4fm_9c8ce093-b801-4390-b647-cd3ea9430ab4/networking-console-plugin/0.log" Sep 08 14:52:39.877538 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:39.877511 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/1.log" Sep 08 14:52:39.884494 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:39.884464 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-tbsgb_7facf738-68e8-45a4-b6a1-6758b93a47ed/console-operator/2.log" Sep 08 14:52:40.594015 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.593973 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql"] Sep 08 14:52:40.599698 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.599664 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.606317 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.606284 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql"] Sep 08 14:52:40.649979 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.649946 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91497141-3d93-4398-b529-34859e5a92c8-podres\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.650179 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.649989 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91497141-3d93-4398-b529-34859e5a92c8-proc\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.650179 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.650027 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91497141-3d93-4398-b529-34859e5a92c8-sys\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.650179 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.650107 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91497141-3d93-4398-b529-34859e5a92c8-lib-modules\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.650179 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.650136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvp94\" (UniqueName: \"kubernetes.io/projected/91497141-3d93-4398-b529-34859e5a92c8-kube-api-access-lvp94\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.751175 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.751133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91497141-3d93-4398-b529-34859e5a92c8-podres\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.751385 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.751196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91497141-3d93-4398-b529-34859e5a92c8-proc\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.751385 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.751274 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91497141-3d93-4398-b529-34859e5a92c8-sys\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.751385 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.751303 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/91497141-3d93-4398-b529-34859e5a92c8-podres\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.751385 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.751323 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/91497141-3d93-4398-b529-34859e5a92c8-proc\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.751385 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.751380 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91497141-3d93-4398-b529-34859e5a92c8-lib-modules\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.751677 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.751406 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/91497141-3d93-4398-b529-34859e5a92c8-sys\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.751677 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.751413 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvp94\" (UniqueName: \"kubernetes.io/projected/91497141-3d93-4398-b529-34859e5a92c8-kube-api-access-lvp94\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.751677 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.751504 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/91497141-3d93-4398-b529-34859e5a92c8-lib-modules\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.759894 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.759862 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvp94\" (UniqueName: \"kubernetes.io/projected/91497141-3d93-4398-b529-34859e5a92c8-kube-api-access-lvp94\") pod \"perf-node-gather-daemonset-hg6ql\" (UID: \"91497141-3d93-4398-b529-34859e5a92c8\") " pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:40.922800 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:40.922683 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:41.235048 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:41.234785 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql"] Sep 08 14:52:41.275441 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:41.275366 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-r2cfl_f3b66b59-514f-4b58-abed-5b1d698c556e/volume-data-source-validator/0.log" Sep 08 14:52:41.729864 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:41.729819 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" event={"ID":"91497141-3d93-4398-b529-34859e5a92c8","Type":"ContainerStarted","Data":"c8d121fe3deda10f842c397d5d5c7687ed3e887d0f07b2a1367585cf64263338"} Sep 08 14:52:41.730348 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:41.729906 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" event={"ID":"91497141-3d93-4398-b529-34859e5a92c8","Type":"ContainerStarted","Data":"29213c69d565c69805f3f3da67a0a4526f77895f3909da99166f6882829de15f"} Sep 08 14:52:41.730348 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:41.729935 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:41.746229 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:41.746161 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" podStartSLOduration=1.746137478 podStartE2EDuration="1.746137478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:52:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:52:41.744143027 +0000 UTC m=+2174.028460141" watchObservedRunningTime="2026-09-08 14:52:41.746137478 +0000 UTC m=+2174.030454593" Sep 08 14:52:42.408285 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:42.408257 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4wnjb_fa59392d-7939-41be-bcb6-0a382dfa8583/dns/0.log" Sep 08 14:52:42.432044 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:42.432016 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4wnjb_fa59392d-7939-41be-bcb6-0a382dfa8583/kube-rbac-proxy/0.log" Sep 08 14:52:42.613126 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:42.613098 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xrx9d_f45cf07b-5cde-44a3-b0e0-2bc6bad77f7c/dns-node-resolver/0.log" Sep 08 14:52:43.160190 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:43.160157 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-77bc4bb45d-c2pxl_b188798f-94d5-4067-9247-8e18472be130/registry/0.log" Sep 08 14:52:43.233690 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:43.233659 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hlpbr_a4d496bc-aa63-48dd-b346-e04b0468e03a/node-ca/0.log" Sep 08 14:52:44.831305 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:44.831248 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jwqsz_07ac3b36-1d40-4ebe-a52f-31237baa182e/istio-proxy/0.log" Sep 08 14:52:44.846663 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:44.846631 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jwqsz_07ac3b36-1d40-4ebe-a52f-31237baa182e/istio-proxy/1.log" Sep 08 14:52:44.887901 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:44.887863 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-kzbbd_a5385bcf-56ea-4679-87b6-44a041f2a606/payload-pre-processing/0.log" Sep 08 14:52:44.932037 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:44.932000 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9xh8f_88bb47cd-a680-4da8-8e34-092fb40d590d/payload-processing/0.log" Sep 08 14:52:45.869575 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:45.869545 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pxs2b_38343840-71de-4d89-b76f-d628180edba4/serve-healthcheck-canary/0.log" Sep 08 14:52:46.376297 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:46.376263 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-fpwss_bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55/insights-operator/0.log" Sep 08 14:52:46.377336 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:46.377309 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-fpwss_bfd3f6ba-5d2f-4300-8fdd-f0dd38904e55/insights-operator/1.log" Sep 08 14:52:46.399468 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:46.399402 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8dx7_91526aea-a23b-4e12-a739-4a4b9c73433d/kube-rbac-proxy/0.log" Sep 08 14:52:46.422907 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:46.422874 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8dx7_91526aea-a23b-4e12-a739-4a4b9c73433d/exporter/0.log" Sep 08 14:52:46.446562 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:46.446534 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8dx7_91526aea-a23b-4e12-a739-4a4b9c73433d/extractor/0.log" Sep 08 14:52:48.757148 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:48.757118 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rt5bz/perf-node-gather-daemonset-hg6ql" Sep 08 14:52:49.075439 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:49.075357 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cb4b89648-l9xvs_a5a1daac-ea5b-4f55-81c3-e9eb4a44351b/manager/0.log" Sep 08 14:52:49.209473 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:49.209446 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-566d57d45b-gwwdv_7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9/manager/0.log" Sep 08 14:52:49.236363 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:49.236330 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-566d57d45b-gwwdv_7510c360-4e6f-4dbf-a68c-9f51ec6f0bd9/copy-manifests/0.log" Sep 08 14:52:50.689236 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:50.689137 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-655586cd5f-sjq9f_09afb555-557d-44da-b8f1-0d387e2b5706/manager/0.log" Sep 08 14:52:50.750489 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:50.750456 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-q2fqk_18616aa8-b8a4-4959-84c8-22e91f878b43/openshift-lws-operator/0.log" Sep 08 14:52:51.019315 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:51.019278 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jwqsz_07ac3b36-1d40-4ebe-a52f-31237baa182e/istio-proxy/0.log" Sep 08 14:52:51.030367 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:51.030335 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jwqsz_07ac3b36-1d40-4ebe-a52f-31237baa182e/istio-proxy/1.log" Sep 08 14:52:51.061864 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:51.061829 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-kzbbd_a5385bcf-56ea-4679-87b6-44a041f2a606/payload-pre-processing/0.log" Sep 08 14:52:51.096590 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:51.096563 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9xh8f_88bb47cd-a680-4da8-8e34-092fb40d590d/payload-processing/0.log" Sep 08 14:52:53.337314 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:53.337278 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jwqsz_07ac3b36-1d40-4ebe-a52f-31237baa182e/istio-proxy/0.log" Sep 08 14:52:53.350434 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:53.350391 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-jwqsz_07ac3b36-1d40-4ebe-a52f-31237baa182e/istio-proxy/1.log" Sep 08 14:52:53.386308 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:53.386279 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-kzbbd_a5385bcf-56ea-4679-87b6-44a041f2a606/payload-pre-processing/0.log" Sep 08 14:52:53.416981 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:53.416885 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-9xh8f_88bb47cd-a680-4da8-8e34-092fb40d590d/payload-processing/0.log" Sep 08 14:52:56.070010 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:56.069981 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-wxm5l_88047337-66c2-44bb-8f97-3a98a11cc754/migrator/0.log" Sep 08 14:52:56.092939 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:56.092915 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-wxm5l_88047337-66c2-44bb-8f97-3a98a11cc754/graceful-termination/0.log" Sep 08 14:52:57.460144 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:57.460116 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kt9z_9d5bf364-23e4-42d1-b576-92465114d3c8/kube-multus-additional-cni-plugins/0.log" Sep 08 14:52:57.484533 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:57.484487 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kt9z_9d5bf364-23e4-42d1-b576-92465114d3c8/egress-router-binary-copy/0.log" Sep 08 14:52:57.506708 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:57.506682 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kt9z_9d5bf364-23e4-42d1-b576-92465114d3c8/cni-plugins/0.log" Sep 08 14:52:57.529681 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:57.529653 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kt9z_9d5bf364-23e4-42d1-b576-92465114d3c8/bond-cni-plugin/0.log" Sep 08 14:52:57.552320 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:57.552294 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kt9z_9d5bf364-23e4-42d1-b576-92465114d3c8/routeoverride-cni/0.log" Sep 08 14:52:57.574373 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:57.574350 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kt9z_9d5bf364-23e4-42d1-b576-92465114d3c8/whereabouts-cni-bincopy/0.log" Sep 08 14:52:57.596244 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:57.596204 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6kt9z_9d5bf364-23e4-42d1-b576-92465114d3c8/whereabouts-cni/0.log" Sep 08 14:52:58.026880 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:58.026848 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wrvcw_13b04066-34fd-477b-895e-fe98f76e661d/kube-multus/0.log" Sep 08 14:52:58.050256 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:58.050232 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-998rf_5267f2dc-393e-40f2-8b78-25179961c1a3/network-metrics-daemon/0.log" Sep 08 14:52:58.069122 ip-10-0-141-155 kubenswrapper[2578]: I0908 14:52:58.069100 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-998rf_5267f2dc-393e-40f2-8b78-25179961c1a3/kube-rbac-proxy/0.log"